explicit length

[Bob Winter]

Hello there,

first of all: I'd like to apologize as I think this topic has been covered enough. I tried to search it up via Google, and got SOME results - but none that really satisfied me. So, if you know some additional references please feel free  to link me to them instead of repeating - it's just something I want to learn about as I seem not to get my head around it.

As I started my "career" into hobbyist programming by some HTML/JavaScript executed in the local browser, then moved along to some PHP and finally ended up stuck to Java for over a decade now, I never had to deal with explicitly providing lengths of fields or data structures - and thank got no allocating or releasing memory or pointer arithmetics. All the languages I used till today do this internally.
But as I currently working on some project actually require me to use some C/++/# I got really confused by the "interface" some "extension" has to provide. It has in its most basic variant three parameters: an output-pointer, some input, and the maximum allowed size for the output-pointer (I'm sorry if I use some terms wrong here - I'm not used to them.). Aside from that really weird order (I would had understand it a lot better if the output buffer and its max length are follow eachother.) why is there a requirement for passing such explicit length/size parameter along in the first place?
Along with that, in the examples, there're a lot of other places where several sizes of input and output buffered are passed along. Not just for strings, which in C are \0 terminated, but for all sorts of things (mostly for those "struct" things - I only understand them this far as they're what I would call a class in java).

My maybe not so simple question is: WHY?
Is C that old that back in the day it was just "cheaper" to pass along a size/length than have it somewhere stored or even computed/checked? I'm so confused by this style of programming it somewhat got me scared - let alone all that stuff possible with all that pointer-arithmetic stuff and all those buffer overflows/underruns. I'm just so used to Java and it does all that for me.

[Tim Holloway]

You only have to pass a length if its something that cannot be known at compile time. But C and is was a lot less aware of lengths than Java is. In Java, the core language protects you from writing beyond the end of an array. Because an array is an intelligent object. But in C, an array is an allocated area of memory and what you actually see of it is only a pointer to that array. So it's very very easy to write not only beyond the end of an array, but even before the beginning of an array!

The storage size of a struct is known at compile time, but not at run time. Structs could often be variable-sized because they contained unions (popular in C) or were subclasses (in C++). A C struct is equivalent to a C++ public class. C++ built its concept of classes from structs.

C has often been called a "high-level assembly language" and like assembly language, what you see is almost the entirety of what you get. Really about the only exception is virtual methods in C++. Otherwise, there is no descriptive data passed around at runtime. This can be very efficient, but you've obviously already seen the price.

C was also designed for multi-user systems with perhaps as little as 8K of RAM per user. Or possible even per machine! It couldn't be made very intelligent because when you don't have enough RAM to do everything at once, you have to make multiple passes and lots of data swaps to disk and other very slow overhead operation. We've reached the point where even cheap cellphones vastly exceed the RAM, processor speed and long-term storage capacity of those days, but the constraints that made the orignal C made it what it is. If you put all the advanced features that other languages of today have, it wouldn't be C anymore. At best it would be javascript. or Java or something similar.

[Bob Winter]

Just another imformative explanation from you Tim, really impressive.
I hearing this "high-level ASM" for the first time as I always thought that C is already classified as a "high level abstract" and to some degree machine independent language. I guess I'm way to used to Java as it's basically the only language I'm using for the past 15 years.
I also already tried to search up about creating dynamic libraries that can be loaded at runtime, but from what I found how to actually design them differs vastly between writing something for the windows platform or some linux distribution and that even to maintain source-code level compatibility one has to stuff a lot of pre-processor stuff in a source file so at the time the compilers pre-processor runs over it the result somewhat automagically (yes, spelling intended) comes out as something that only consist of platform specific source which then gets compiled - very confusing.
Luckly what I want to implement doesn't seem all that complicated (although I still wonder if it may be possible to cross-compile a windows binary on a linux system as I don'T really want to clutter up my system of several gigabytes of dev stuff just to compile a few lines into a small lib) but still maybe up for some topic on its own:
As Java offers the JNI which also allows full-duplex bi-directional communication I'd like to (if possible) implement a small lib which just takes the string given by the main application - calls some java code to process it - and sends back the result. But, from as far as I already read up: just connecting two applications by a single lib doesn't seem to be that easy straight forward possible as I hope to as the pointer only stored in the memory spaces of the respective processes. One of the techniques to get around this seem to be things like unix domain sockets, regular ip sockets and even memory mapped files and using some sort of os level event bus system to signal the other program that there new data in the memory to process - isn't there a simpler way?
I also once watched a video on youtube showing the very basics of whats actually inside the java bootstrap binary - and I guess this can also be done within the lib loaded by the main application (so it basically runs a VM inside its process sapce). That would be cool by me as I then still can develop my logic in java and just have it run by the vm created by the main application instead of its own instance.
One may see from as how my text turns into getting harder and harder to read I'm getting more and more confused as on how to get started.

Maybe, for anyone still with me and may willing to give me a help in the right direction, I shall give you some more specific information about what I want to implement:

As I currently working on design a modification for the game Arma3 I face the challenge to implement some way to connect to a database. The current way is to use a lib known as "extDB3". It's basically a wrapper for the mysql-connector/c connecting it with the game using its "extension" api: https://community.bistudio.com/wiki/Extensions
They way this is currently achieved seem to be this:
The developer of the extDB3 lib provides two modes of operation: raw SQL and prepared statements defined via an INI loaded at first time the lib gets initialized. As the extension api of arma3 has a limit on about how much data can be send back from the extension to the game engine (currently about 10kB) there can be results bigger than this which has to be transfered in several consecutive calls to the extension. To make this possible the extDB3 extension buffers the result it gets from the database in what looks like a map along with an id which gets returned upon the initial a-synchronous request and it's up to the arma3 code to keep polling for the data until an empty string is returned to mark the end (I guess that's one example of unknown length at runtime which justify an explicit length/size information).

The main issue I currently face is that most of the code I have to deal with is already several years old (the extDB3 lib was last modified back in 2017 and the base for my own new mission is from about 2014-2017) - and in the mean time the arma3 game engine has evolved so my goal is a complete rewrite to take a) advantage of all the new features and b) make it as performant as possible as the current version really shows how slow it all is when you have more than just a few players interacting with each other and the world which, although extDB3 is pooled and multi-threaded, can add up to some lagg as the main game engine itself (and its multiplayer server) is just one big single threaded mess.
To release the main server instance of it's heavy load the arma3 game provides what they call "headless clients". It's basically just another game client but without the graphical overhead - just the logic itself. The benefit one can get from this is to offload lot'S of work from the main server instance to these headless client instances. This way I managed to get down the overall system usage of one single server instance of about 85% to just about 30% with a second client insatnce with about 20%. Not just I got down the usage but also can now take advantage of the two physical cpus the server has to split of the load between them. My goal with all the rewrite of the game mission and the extension and the database stuff is to maybe split the workload even further to take more advantage of the available system resources (2 quad core cpus) and to hopefully increase the performance even if the server fills up with players (currently I set a max of about 175 players - which the engine in fact do can handle (seen it other servers with 150+ players at the same time - as the staff also had a lot of work done on their own it was quite performant)).

Why do I post this hear instead of the game forum? Well, the way I see it this is a programming "problem" which is best solved in a programmers forum. The game forum for arma3 doesn'T go that deep into programming but only into using the provided tools with the game - which ends about the point where a call is made from the game to the extension. So I doubt I can expect much help over there.

To summarize: What I'm looking for is a way to either connect another process with my own JVM via a dynamic library - or, if that'S not possible, how to start up and run a JVM within the space of another programm by implement its startup code in said dynamic library - to, in the end, be able to develop my extension in java and only use the C lib as communication binding as the arma3 game engine only provides this one option to talk to external code.

Any help and advise are appreciated. Thanks in advance.

[Tim Holloway]

Wow. That's a lot to digest. Let's see if I can hit the major points without forgetting something.

Dynamically loadable modules. In Windows, these are Dynamic Link Libraries (DLLs). In Unix-like systems, they are Shared Objects (.so's). Unix .so files may be versioned, which is something that DLLs don't support. Windows COM objects are often implemented in DLLs. Don't ask me about .Net assemblies, as it has been a Loooong time since I did any .Net and never spent that much time on it. Plus from what I've seen, DLLs are alive and well in Windows even now.

The concept for both windows and unix libraries is similar but not identical. Especially the process of taking a bunch of object modules and wrapping them up in a library. And of course, the object file formats themselves are quite different.

But there are toolkits that can be used to make code that will run under both Linux and Windows. Several of them, in fact. The most ambitious ones include abstract OS-independent GUI APIs and that's generally what's used on major open-source projects that can run under both OS's. Stuff like Gnucash and FreeCAD. Those apps are way too large to maintain separate codebases for multiple OS's. I can build an entire working Linux kernel in less time that it takes to build FreeCAD.

I don't really recommend using JNI unless you absolutely have to. Since Java is mostly complete as long as you stay aware from programming direct-to-hardware and/or specialized OS services, most JNI code is actually just glue to a DLL or the like.

Headless operation. Both Java and Windows often steal graphics support from the system's video card and OS graphics services. One reason why Word documents can print so differently on different machines is because it does all its typesetting operations by consulting the machine's internal font functions and using the systems installed fonts for character metrics. It's not as noticeable when you're using softfonts like TrueType, but originally, you had to work with the fonts that were hard-coded inside the particular brand and model of printer you had plugged in and if there wasn't a particular font on a printer, Windows would pick the best match from what it had available and re-render the page accordingly. Including changing where the page breaks were.

But it's a waste of resources to plug in a monitor and launch a GUI desktop for a machine that's crowded in a locked server room just so you can render pie charts on the fly in web pages. So there's a headless option for Java. For non-java apps, it's often possible to get the same effect by calling a graphics library directly. In Windows, I believe that all the good stuff was originally in "graphics.dll". On the Amiga, there were actually several graphics layers each in its own library. One, for example, dealt primarily with handling rectangular graphic objects and supporting the ability to move them around, resize them, and - most importantly - stack them over each other. It was, in fact, called the layer library and you can see where someone designing their own GUI desktop would appreciate it.

[Bob Winter]

Although not as helpful, but thanks for crunching through the wall of text and reply anyway.