[UX] Pass configuration as file #923
Comments
|
Or to put it differently: “Simple things should be simple, complex things should be possible.” ― Alan Kay |
|
I like the idea of using a JSON file as its format can be largely the same as the REST API. I understand why the REST API is there, but for development it seems that everyone has some scripts around to start firecracker with curl invocations. I probably have three variations. |
I would be really interested in the rationale for why the initial VM configuration is designed this way. From my current understanding, I think this adds lots of complexity on both the user and Firecracker's side of things. |
raduweiss
added
Feature: API
Roadmap: New Request
Priority: Medium
|
I did some crude measurements and from starting firecracker to actually running the VM (i.e. getting a response to the InstanceStart request), it takes more than 30ms. 2ms are waiting for the API socket to appear and the rest is for the 5 requests via the |
|
Hi, I have just submitted an RFC pull request that provides a rough proposal implementation that works. I have experimented with it a bit and on my 5-years old MacBook I am able to start and complete new VM within 20ms. However for some reason exiting new executable process takes additional 30-40ms. Please see output of strace: As you can see it started at 09:37:08.951057 and exit_group() was called at 09:37:08.974119 (23 milliseconds in this case) but per last line What could be a cause of it? As far as I understand exit_group() terminates all threads and in this case there is one vCPU thread apart from the main one. But I do not understand why it would take so long for this thread to terminate. Any ideas? Where is a flaw in my implementation? |
|
@wkozaczuk Very nice! One comment: strace is usually not a good way to measure time, because the system call tracing it uses is very inefficient. So likely the performance is better than what you measured above! For measurements like these, it's better to have the application itself generate timestamps and log them.
Wild guess: Maybe it only counts as fully destroyed once the parent process fetches the exit status via |
|
Since my last comments I have made more improvements to my "command-line" prototype of firecracker with aim to improve startup and shutdown time. The shutdown part is really where I think the crux of the problem lies. I ended up creating another branch (for those interested - https://github.com/wkozaczuk/firecracker/tree/sockless_optimize, please see the diff between this and older sockless branch). I mostly followed @blitz's advice to make vcpu thread (child thread) end orderly and main parent thread join it instead of calling exit() on guest shutdown request. In general I managed to shave off around 10ms but there is still 20ms wasted somehow after the process terminates as reported per time. Here are the average times from the 25-times run of firecracker launching simple OSv hello world example with single vCPU back-to-back: As you can see on average it took:
So there are 2 questions outstanding:
Regarding 1) it might be useful to look at this fragment of strace output: These I think are "new" (comparing to the output I reported 2 weeks ago) system call that come of firecracker_cmd orderly closing all (?) file descriptors. I am new to Rust nor I understand firecracker code well, but I think it comes from one one of the drop() method (destructors) possibly triggered by destruction of the Vmm object? EpollContext? As far as the 2) question goes I have no idea what happens there. There must be some other resources Linux needs to close when handling implicit exit(). Any other ideas of what these could be? Some wild guess - is it because of seccomp? |
|
A few weeks ago we had a design discussion about passing a configuration file instead of using API calls to configure the initial state of the microVM. Passing the configurationFor passing the initial configuration we have a few options as follows:
Running Firecracker without the APIRunning Firecracker without the API should be possible, but the default will still be having the API server. We have two options here:
Some other requirements
Deserializing the configurationFor deserializing the configuration my proposal would be to define a use serde::Deserialize;
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
struct VmmConfig {
boot_source: BootSourceConfig,
block_devices: BlockDeviceConfigs,
network_devices: NetwordDeviceConfigs,
...
}The current configuration files do not derive Serde deserialize at the moment, so this also needs to be changed. CC: @firecracker-microvm/compute-capsule please let me know if you have any feedback/other comments. |
|
Hi! I like the STDIN configuration best. We can add a new parameter which represents that configuration data should be read from stdin. I guess we'll have to extend this over to the jailer as well. Regarding Running Firecracker without the API I'd go with the second option for now. It looks difficult to cleanly separate the API server from the I like the bit about having separate API and VMM structures (at least where they wouldn't overlap perfectly), and it's something we should pursue. However, for the time being, is there anything wrong with using a composition of the current API structures? |
Do you mean that we should only read the configuration from stdin when that flag is present? Or that we should support both file and stdin? I don't want to have them both honestly as it complicates the code for no good reason. I think we should stick to one (either file or stdin).
Why do you say that?
I think the problem is adding more technical dept, which I would very much like to avoid. To workaround this issue I think we can prioritize the aforementioned refactoring and block this issue till that is done. |
|
@andreeaflorescu Thanks for the very nice write-up. It really helps. I am definitely in favor of "Configuration via STDIN" - most elegant and flexible. As far as "Running Firecracker without API", I am in favor of "Conditional compilation", mostly because of the smaller executable and fewer dependencies. Also, it feels to me that somehow "Passing the configuration" and "Running Firecracker without API" deserve to be separate issues (though very related), no? Also, I was wondering if the potential implementation should be part of a single PR or many smaller ones? If many, in what order/way we should divide it to allow for best coding and reviewing experience? |
|
@wkozaczuk I agree with you and running Firecracker without the API should be a separate issue. We can create a few more issues related to this one and "convert" this one to a story (aka we can track the progress of all other issues in this one). If we have multiple PRs is far more easier to review and the chance of getting them merged fast is higher. I will think about a split that actually makes sense. But before that we should decide if we want to include the refactoring related to separate structures for VMM and API. I am inclined to say yes as I am afraid that otherwise the code will be a bit spaghetti and parts of it will be thrown away later. CC: @firecracker-microvm/compute-capsule please take a look at the suggestions. |
|
Configuration via STDIN (using same structs/api as when configuring through unix-socket) is my preferred choice as well. Regarding the api-server when running with stdin input I am leaning towards option 2 of simply not initializing/starting the api thread. Conditional compilation brings in extra complexity on the testing side and less versatility for customers. I don't see the argument of smaller attack surface as particularly strong in this case. As for separating resources structs into descriptor structs and functional structs, it would be an improvement, but I don't believe it necessary. Would be great if the person tackling this story would also tackle this particular topic, but I wouldn't make it a requirement. |
|
|
It looks like we have an agreement regarding file vs stdin and we want to move forward with stdin. As this is not necessarily related to the other issues, I propose tracking them in separate issues and resolving them in separate PRs. For this purpose I've opened two issues:
I think we should postpone the decision of using conditional compilation vs soft disable for now. Our API server is currently rather complex and brings in a lot of dependencies (last time I checked there were ~70 transient dependencies). It would be interesting to know how does the complexity of the api_server translates into memory and build time overhead. But, since we are trying to implement our custom api_server and thus removing hyper and tokio, I would say to take a decision after we start working on #615. |
|
My concern about the STDIN variant is that parsing JSON out of a stream is generally error-prone: if the document is badly formatted you might not be able to tell easily (eg. missing Also, from a usability standpoint:
Is there are any reason not to support both files and STDIN? This shouldn't be very complex to implement. |
|
Hi everyone! In light of @petreeftime's comment, and thinking a bit more about the different implications of STDIN vs other ways of passing parameters, how about actually using a single command line parameter to pass the entire JSON content (for example We mentioned using command line parameters for configuration passing, but I think we were still reminiscing about the old way of having different parameters for different VM configuration options (such as vCPU number, memory size, etc.), and not simply using the value of a single parameter to hold the entire JSON config. Also, one problem of the STDIN approach that I totally forgot about initially is that the Firecracker process created by the jailer has STDIN bound to |
raduweiss
changed the title
raduweiss
added
Roadmap: Tracked
|
Solved by #1212. |
raduweiss
removed
the
Priority: Medium
At the moment, Firecracker is configured by sending API requests via a
AF_LOCALsocket that Firecracker creates. This means that a user of Firecracker has to:This is unfortunate, because it requires quite a bit of code and a lot of synchronous and stateful interaction just to get a basic VM going. As an alternative, I would propose handing in the static configuration of a VM (vCPU configuration, disks, network devices, ...) as a JSON file that is provided to Firecracker when it is started (via a file or via stdin or if it's small via the command line). Firecracker then requires no additional interaction from starting the process to running guest code.
This relates to #343.
The text was updated successfully, but these errors were encountered: