Someone Was Asking About Devops …
Someone I know was asking me about devops the other day, particularly the number and variety of tools and how to integrate them …
I tend to do things a bit differently than most people, because I find it both more efficient as a developer and more maintainable as a sysadmin. Since I’m working by myself on my current project, and have done on a number of projects, certain things I would do in a team development situation don’t apply, but I’ll detail first how I do devops on a single person project, and then how I would modify that for a team project.
There are two seemingly contradictory principles in tool development:
1. Use the right tool for the job — there’s no magic hammer, not everything is a nail.
2. Use a consistent methodology for every task.
The first is the reason for the number and variety of devops tools, and the reason tools like Maven are inherently a headache.
The second is the reason headaches like Maven exist. The types of tools commonly used in devops don’t lend themselves to #2 unless #1 is compromised, and for a specific, simple reason, which I’ll get to in summarizing what makes my own personal setup simpler and easier for me.
When I complain about the lack of consistency in GIT, referencing rule #2, I’m not referring to the plethora of GUI GIT tools, but to the inconsistencies between command switches in the command line interface itself.
Although most of the tools will be known to most involved with devops — GIT, Maven (but for one thing only, not for any of the other things it can do), Hudson/Jenkins, etc. — very few setups are or can be this simple.
The code and runtime for the current project is in two parts, each in its own container, running in the same OS or virtual OS instance (or one Docker instance, it doesn’t matter so long as it has sufficient resources). The containers are:
- Pharo: Pharo contains the full open source Smalltalk language, development environment and VM.
- A JVM and an OSGi runtime container with custom Java code and its dependencies. In this particular case I’m using Equinox, which is part of Eclipse (the base of Eclipse in fact), but I could swap that out for Apache Felix, Knoplerfish or even a micro-container like Concierge/Kura (designed primarily for Raspberry PI deployments).
Smalltalk is written , including the interpreter and JIT compiler that compiles it directly to assembler, and all of that code is live in the development and runtime image. There’s no such thing as a separate “command line” in Smalltalk development (there is a workspace/playground and transcript, which have command functionality and output, but they run inside the VM, and Smalltalk code can be called from the command line, but it still runs in the VM).
The ‘image’, which is properly both the development and platform independent portion of the complete runtime VM, is one file. The whole image, which contains the language, JIT, tools and libraries (some of which may add to the tools available) is loaded and all classes are live objects, not templates for objects as they largely are in C++ and as they at least appear to be Java (in reality they aren’t, which is one of the numerous discrepancies between the semantics of Java and its implementation).
This means that the image, when stored, maintains the complete state of the VM. If I save the image with a Zinc streaming web server running on port 8088, when I load it, the server is restored to its running state on port 8088.
Although Smalltalk can’t be developed in a text editor, since there are no text files that store source — it’s all in the image, there are ZeroConf scripts to deploy it. The versioning tool that works with code repositories in Pharo (and Squeak), is called Monticello. The package manager (somewhat like Maven, though more in a certain sense like NPM from NodeJS, in that it pulls source, not built artifacts, the only built artifact is the image), is called Metacello. As well, any class can be called from a running image via bash (or cmd on Windows) if it understands the HandleCommandLine message, and that class can call any of the Metacello classes to handle package management. So, unlike most languages, where the entry point is a single function or method such as main (Args args) {}, any Smalltalk class can implement methods to be called if the HandleCommandLine message is passed to it.
In concrete terms, the base Pharo VM bootstrapper and Image (which constitute the runtime) can be installed via bash or cmd using wget or curl –L like this (on Linux — on Windows you just replace the | bash with | cmd):
curl –L http://get.pharo.org/60+vm | bash
That downloads the latest (6.0) image and the latest bootstrapper.
The entire image can be run in the VM, including all the development tools. Since I would be using the headless VM, the GUI classes would be irrelevant, and virtually all Pharo development tools have GUI elements. The advantage of doing that, although it uses a bit more memory, is that in the event of a runtime issue the debugger and all the code analysis tools, which are better than those available in any other language, are there to fix the issue, I just need to start up the same image with a regular VM rather than the headless VM. If writing a GUI app, the VM can be locked down so that the tools are inaccessible. If memory is more of an issue, packages that are not a necessary part of the runtime can be unloaded.
When I say more memory, I’m talking about ~100–200MB for the entire thing, with all the development tools and libraries and GUI. Since in the case of my current project the data could be 50x that, saving 30 or 40MB by dumping the development tools isn’t worthwhile. With a minimal Smalltalk image, for instance the classes necessary to run, monitor, log and manage microservices, the runtime could easily be brought under 40MB, and would use less memory for a set of a dozen services than a single Java or JS microservice would use on its own.
Back to devops. Although Monticello and Metacello are Smalltalk specific, the server those clients use for source and configuration management can be a relational database such as PostgreSQL or Oracle, a regular file system, a GIT server, a local GIT repository, or any number of other back ends. Monticello versions by a number of means — author of the latest commit, absolute version number, or symbolic version, or any combination of those. It also versioning to the method level, not only to the class level. Loading a version always loads as source code, if there are conflicts, they need to be merged or one or the other chosen, since the runtime has to always be consistent, but merging is fairly painless even if there are conflicts, because the diff/merge tools and other related browsers are first class.
The Configuration class, which by convention is implemented as ConfigurationOfWhatever (for instance ConfigurationOfSeaside3 loads and configures the best-known Smalltalk web application framework and server), can load the application and include as many other Configuration classes as are required to load any dependencies. Thus, once the curl or wget gets the base image, the next command in the install script will look something like this:
./pharo ConfigurationOfProject version: #development; load
This will load my code, all the dependencies it uses from whatever repository is specified in the configuration class, and the state the application was in when it was committed. I then call another configuration class, but this one gets the Java code and instantiates the other language VM.
The reason for using Pharo in the first place wasn’t just the ease of deployment, although that factored, the reason was the need for concurrent speed in handling and parsing text streams, specifically XML streams. On some of the high-end aggregation service routers we are using, Cisco has deprecated the old command set (with the rather arbitrary responses) in favor of a completely XML based API. The XML requests and responses are sent over an SSH tunnel to a specific port (which is configured in the router from the regular SSH command shell).
Five minutes of events, rendered as XML, can result in 100+MB of XML, and the JVM can’t easily handle that kind of text data volume, whether the code is written in Java, Scala, Clojure or any other JVM language. Java itself is useful since code to connect to the overall project database in DB2, queues for MQ series etc. can be generated and configured automatically, but the initial data needs to be filtered by a faster means.
To give you an idea of the speed of Pharo compared with Java, rendering a full web page with JQuery widgets, a Twitter bootstrap CSS based UI, etc., exactly what the Portal my project integrates with, already uses, can be done in less than 50ms. Initially starting up the application server to do that takes 113ms on an seriously underpowered laptop. The Xtreams library gives a streaming interface with event generation not only on streaming data such as XML responses, but on anything — collections, database queries, JMS queues etc., and it’s also very fast. It can handle that 100+MB of data as fast as a gigabit Ethernet connection can transmit it, while only using about 6% of the dual core CPU on my work laptop. Since I need 8 of those Gigabit ports to handle the total volume, JVM languages are unworkable.
Cisco themselves use Perl due to the text processing requirements — they provide no Java, WS API or REST API, but although Perl is very fast at text handling it’s also nearly unmanageable, since there’s no defined application, just a collection of (for all practical purposes) unrelated scripts in text files, and no state maintained between the individual scripts.
Finishing with Pharo, since we have that configured and running with just the two lines above in a bash or bat script (from a devops perspective, the code that makes those scripts work is inside the black box of the Pharo image), and all the configuration that it’s loading can simply be committed as source in a GIT repository, the portions of the functionality handled in Pharo are:
1. Calling an XML API by sending a request over SSH, waiting for the response, and then streaming the response while the parser parses each StAX event.
2. Filtering the response data to only the data points we actually need for the UI.
3. Storing the data in a base 64 encoded object DB, which doubles as both an archive and a query store that can be queried using XPath queries.
4. Configuring the JQuery widgets that provide the dynamic portions of the Portal page (in most of Portal, this is done by endless redirection which finally ends at some JavaScript code, in the case of IP Monitoring, while the JSP is pretty much identical to the others, there is only one level of indirection, which calls the Pharo web server to retrieve the configured widgets (tables mainly, but a few others specified in the wireframes).
5. A microservice which exposes the data for a specific widget as a REST endpoint, so that the widget can get and update its data, and respond to new data events via a JQuery event (so that if the page is left up, it will automatically refresh when new data arrives).
6. A microservice manager that will shut down the microservice if a small JS script included with the widget, which is set to renew a lease every n seconds, fails to renew within n x 4 seconds, in order to return the resources to the VM if the user is no longer on the page.
7. An MoM client to push the filtered data to the Synapse micro-ESB in the Java OSGi container.
8. Configuration and monitoring of the OSGi container, including restarting it if necessary.
Essentially each container is instrumented in a manner that they can check each other’s operating state is within predefined limits, and recycle the other if needed, and check that the OS is operating within reasonable limits and doesn’t need recycling. Since Pharo can be scripted to store its full system state on shutdown, when it restarts the system state is precisely what it was at the moment of shutdown. As a result, it requires no external configuration, and since it provides the configuration to the OSGi container, it needs no configuration files etc. that are difficult to update OTN.
As well as configuring itself via a Metacello configuration script, the Pharo VM has a class that loads and configures the OSGi container and code. This class first uses wget or curl to pull the required Equinox version from downloads.eclipse.org; then it spawns a process and starts the Equinox container, which is configured with Eclipse Aether. Aether pulls the required built jar artifacts for the full Java runtime from Nexus (Maven built artifact repo, essentially any Maven repository, although our own jars will be in our own Nexus), and calls the initialize method on the parent bundle. Since OSGi doesn’t store the system state, the initialize method in the bundle configures the runtime state — connecting to CMDB, starting up Synapse with the configured queues and topics, opening a local socket connection for Pharo to push data into a Synapse queue, and listening for OSGi configuration manager events.
This is the only way in which Maven is used: to deploy built Java artifacts to a Nexus repository where they can be accessed via Eclipse Aether. It’s not used as a package manager, nor as a build tool.
The right tool for every job.
Once the OSGi runtime is initialized Pharo sends some OSGi configuration manager events to it (which are also XML documents sent to a specific port, very similar to the Cisco XML API mechanism) to fully configure dynamic runtime state that can’t be coded in advance in the initialize method.
The OSGi runtime is then responsible for:
1. Maintaining a set of queues and topics for MQ series to pull data.
2. Pushing specific data needed for correlation with other subprojects into the overall project database.
3. Monitor the instrumentation on the Pharo VM to ensure operating parameters are within a predefined set of acceptable limits (for resources etc.). If needed restart the Pharo VM, or the entire OS, in order to free any resources that are being used up.
5. Responding to configuration manager events sent either by Pharo or by the management console to modify the configuration, update to new code or update library dependencies, etc.
That’s basically it. I told you it’s quite simple. I also said that the reason that kind of simplicity is not often possible is simple. That reason is the lack of a consistent GUI to integrate the specific tools.
While development environments try very hard to integrate devops tools, if the result works, it usually only works for developers intimately familiar with those environments. And those environments tend to be very complex and as a result often somewhat large and unwieldy.
Yet Pharo is not much bigger than an average text editor, particularly when the size is compared with machine sizes and bandwidth availability. Why does Eclipse need to be so big? And Netbeans or IdeaJ are not as much smaller as one might expect, since they don’t even give you a live object environment, which at the very least a properly configured Eclipse can.
Writing code, like anything else, can be done best with good tools. Going back to a text editor like Emacs (unless you’re writing LISP) means you give up any number of available tools. The reason you either get the size and complexity of Eclipse, or the size if not complexity of Netbeans or IdeaJ, or few to no language specific tools at all, is a mistaken notion that files are a language construct in programmking languages.
The reason I’m insistent, in Java, of using Eclipse as a live object environment based on OSGi, and using an OSGi based runtime, is that it gives a consistency between the development/test environment and runtime, down to the classes loaded at any point in the code path. OSGi bundling removes any packages not used by the application code from JAR dependencies, just as the Smalltalk image does, although Smalltalk is reflexive enough to do so at runtime. Not only is the environment the same (and very controlled) in both, the actual system state at any point in the code path as far as other packages and classes in the runtime are the same (since classes not reasonably close in a potential future code path aren’t JIT compiled, they don’t exist at that moment in the runtime, and unused classes are not loaded).
The result is higher performance in Java, partly because the runtime uses less memory, but more because each bundle JAR contains all the required classes, they are all loaded (whether or not the bytecode is compiled completely or JIT depends on whether the application is a Java EE application or a regular Java server application), and no others are loaded. This ensures that the runtime is identical to the development / test environment if the application is developed in Eclipse (and Eclipse is properly configured) and ensures that the runtime contains precisely the required classes, while making classpath scans to find classes newly instantiated unnecessary.
The last is both a matter of performance and stability. Since the JVM can only scan the classpath sequentially, opening each compressed JAR file to find the classes it’s looking for, it’s a very slow process, and resource constraints or transient file system/zip handling errors can crash a running JVM.
Eclipse was intended and designed to be a live object environment like Smalltalk, specifically because its predecessor, VisualAge for Java, was written in Smalltalk (as were VisualAge for Smalltalk and VisualAge for C++, there was even a VisualAge for COBOL, also written in Smalltalk). Since Smalltalk is designed from the ground up to be a live object development environment, it both works more smoothly that way and is far less complex.
For some godforsaken reason Java programmers, most of whom were initially ex-C programmers, were unhappy with not having text files to manipulate on the file system, even though a file is not a language construct in any of those languages (it’s an environmental, OS dependent external representation, not one built in to the languages). IBM started the Eclipse project (initially as IBM Workbench) to produce the same kind of environment but using Java to write it and allowing external modifications to text files within the workspace, which essentially replaces the image. This causes all sorts of additional complexity and massively greater resource usage to ensure constant synchronization between the file system representation and the in-memory representation (Eclipse loads the entire workspace into memory and does all operations in memory, just as Smalltalk does).
There is one well known environment that doesn’t use a traditional file system, but instead uses DB2, the AS/400 (or I-Series as it’s been renamed). While it’s not exactly mainstream, its success in its market can be gauged by what its introduction did to DEC, at the time the largest manufacturer of systems in that market.
Smalltalk is niche, and partly because since the language itself and the runtime can be dynamically modified, it’s both extremely powerful and a screw up can do plenty of damage (though, of course, that can be mitigated). You can easily shoot yourself in the foot writing code in any language, but in Smalltalk, as in LISP, Forth and Haskell, you don’t just shoot yourself in the foot, you blow both your legs off.
On the other hand, when you write it properly it works reliably, consistently, and is rarely affected by the deployment environment.
It is niche, but like the other languages I mentioned it has a strange niche. It’s not industry-specific in the way ERLANG is; it’s not functionality dependent in the way FORTRAN is; it’s not even generation dependent in the way C is and Java are, though in opposite ways — you can write higher level code than Java and lower level code than C, since the JIT is itself written in Smalltalk and thus also live and modifiable at runtime. All those languages work, via different structural features, in a similar way, because they’re all written in themselves, other than a small initial bootstrapper that can be written in anything.
When you use Eclipse and OSGi to produce a similar environment with Java, you don’t gain equivalent additional power, but you do gain stability and predictability in terms of the runtime behaving the same way as it did at development and test time. The massive ecosystem of available Java libraries is the only reason the hassle of maintaining that environment is worthwhile. Some of them even work reliably.
A live object environment and corresponding runtime is simply more productive. As soon as you make a change to code, the effects are there in the environment. There’s no need, if everything is configured properly, to restart server processes or go through any sort of write / compile / build / test / debug cycle, so testing each code change is much quicker. VisualAge C++ even managed to make that paradigm work with C++ code.
Code changes are compiled incrementally (and almost instantly) in memory, and any code that the changed code modifies is also recompiled in memory, so side effects are immediately noticeable. As well, tools that check all the object relations, code paths etc. can be written relatively easily. You won’t find the kind of code analysis tools, refactoring tools etc. that are standard in Smalltalk (the main ones are part of the language specification) in virtually any other environment, except to the degree possible in Java with Eclipse, and that’s a fairly small degree.
A team environment would of course affect a few things in the process outlined above:
1. Depending on the team size, a team can write more code, and integration becomes more time consuming. Build servers like Hudson/Jenkins can assist in this by running a full build on every code commit, including running all unit tests (the notion of unit testing comes from Smalltalk). Both Smalltalk and the Eclipse build system support headless building on Hudson/Jenkins, in Eclipse you export and commit the headless Eclipse build libraries. The following is a Jenkins build configuration in Pharo Smalltalk, this particular one for the open source libraries to write XML:
# Jenkins puts all the params after a / in the job name as well:
(export JOB_NAME=`dirname $JOB_NAME` )
wget — quiet -O — get.pharo.org/$PHARO+$VM | bash ./pharo Pharo.image
save $JOB_NAME — delete-old./pharo $JOB_NAME.image — version > version.txt
REPO=http://smalltalkhub.com/mc/PharoExtras/$JOB_NAME/main ./pharo $JOB_NAME.image
config $REPO ConfigurationOf$JOB_NAME — install= $VERSION — group=’Tests’
./pharo $JOB_NAME.image test — sunit-xml-output “XML-Writer-. * “
zip -r $JOB_NAME.zip $JOB_NAME.image $JOB_NAME.changes
Although this is a Jenkins script, because it’s configuring a Smalltalk build it reads rather like Smalltalk code, which is based on message and object reference passing. That’s because Jenkins is not a build tool, it’s a runner of build tools. How the script will look is dependent on the build tool its running, which in turn will depend largely on the language the code it’s building is written in.
Ideally, those specifics would be left to each developer who writes that code, and the uniformity of access would be guaranteed by the UI. But the insistence on working at the file level, as if files were relevant to a developer any more than sockets and other OS dependent specifics, results in having configuration that reads like code, and worse reads like the specific code used in each different job. As it happens, it’s easy in Smalltalk to output such a script, so the build engineer doesn’t have to read it, he just has to run it and check for problems in the output.
As with other tools integrated into Pharo, there is a browser for Hudson/Jenkins that works in a similar manner to all the other Pharo browsers. Encapsulation is not just a language construct, it’s what allows any appropriate division of effort in general. An appropriate division of effort produces a more efficient system, in software development as much as car manufacture.
Going back to the two principles that seemingly contradict one another, the solution, as was seen by the developers of Smalltalk very early on, is to use the right tool for the job, but to access the tool via a consistent interface, whether GUI or text mode.
The hyper focus on files and doing everything via files inherently breaks that means of hiding differing functions behind coherent interfaces, and in many languages prevents writing a coherent interface in the first place, or makes doing so such a complex and demanding task that it will be rarely attempted in such languages.
�p�~j�)
