The ML collaboration tool you’ll want to ride solo.
I’ll admit it. I am a gushing fan of this new product from Allegro AI called Allegro Trains. I’m not sure what to call it — what noun I should attach to this creature. “Framework” and “Platform” have become, to my ears, rather meaningless jargon designed to detach suit-wearing types from their money.
“Harness” is close.
Scaffold. That’s it. A structure you put up before you start building. With a scaffold you don’t fumble with ladders. You stroll the full perimeter of your structure at 100 feet up. You walk supplies right into the 10th floor window. If you drop a wrench it falls neatly at your feet instead of killing your foreman far below.
Trains is an AI scaffold. One that I really like.
And that’s a little peculiar. A bit more about me : I write a lot, especially in the field of software engineering and Machine Learning. I’ve been at it a long time, venting my unfiltered opinions for a semi-large, endlessly tolerant following of readers on Quora: Chris Reiss on Quora.
I’m not exactly known for, um … liking things. I have, over the years and often at length, expressed my distaste for a great many topics.
“Hate” is such a strong word. So let me use that. I’ve hated on, I think, 96.4% of everything I’ve seen. I’ve hated on C++, for example. This isn’t the cursory hatred that one develops after two years of use. By way of bona fides, my C++ skills were developed over 15 years, honed finely enough that I have contributed to the WebKit rendering engine powering Chrome and Safari. This is a seasoned, deeply anchored hatred.
I’ve hated on Design Patterns. On Object Oriented programming. On Agile. I’ve hated on virtually everything Microsoft ever did. I’ve hated on Google and especially on Android and on Tensorflow 1.0 .
I couldn’t wait for the release of Tensorflow 2.0 so I could begin hating on that too.
I say all this only by way of convincing you that I don’t generally like things. So when I finally do, there’s a good chance it’s worth your attention.
It Clicks
There is a story often retold about Steve Jobs. It’s supposed to typify his prickliness, his obsession with minorest of details. It’s the night before the launch of the iPod. Everything is set to go. Engineers watch nervously as Jobs fiddles with his iPod, pacing through a rehearsal of his presentation.
He stops. The headphone jack doesn’t “feel right” when you plug in the headphones. It doesn’t offer a satisfying click.
Jobs wants it fixed. Not his, but the 400 going out to audience members at 9 AM. All of them. The engineers got no sleep, but the audience (and later users) got their click.
To view this as mere obsession to detail is to misunderstand what separates a great design from a merely functional one. The ‘click’ isn’t about the mere task at hand — connecting the headphones. It is about the relationship of owner to object.
Sure, the iPod plays music. But it’s much more than that. It waits with you at the airport. It hopes you both get a chance to impress that gorgeous barista at the coffee shop. It walks with you in the rain.
Jobs got that. That deep relationship. On the engineer’s CAD screen, the jack connects to the headphones. But in its relationship with a human, it goes click. You can hear it in a quiet room and feel it in a loud room. The click says not just “connected” but also, “Yes, that’s the right kind of headphone jack”, “it’s not bent” and “I’m good as new”.
But most of all, it says, “OK! Let’s go!” Jobs understood the importance of saying that. How a beautiful design goes click.
As a Machine Learning researcher who honestly couldn’t possibly care less about a “Platform” or “Framework” — let me describe my conversion to Allegro Trains, one click at a time.
“Leave me alone, or I shall taunt you a second time.”
I’m a lone researcher in Machine Learning. Like so many others, I’m not particularly bound to any one area, and hop the fence from NLP to image recognition, robotics, and so on.
And like so many of us, a lot of my work is very ambitious. I am drawn to the prospect that there is enormous potential for progress, not in the last decimal places of the hyperparameters to enormous pretrained models like BERT, but rather at the simplest, most fundamental level. I’m looking for new architectural possibilities, new approaches.
So my head is all over the place. For every one thing I’m working on, there’s two I should be working on instead — one I forgot to finish and another I forgot to start.
So when somebody wants me to try some new Machine Learning management thingy, my initial reaction is “I work alone, I don’t care, and now there are three things I’m supposed to be doing. Go away. Stay that way.”
Trains gets this. “It’s cool. Just do this, nothing more, and I’ll leave you alone.”
>pip install trains
from trains import Task
task = Task.init(project_name=’Project’, task_name=’Tryit’)
“Now, in your epoch loop, grab the validation metric and do this.”
logger.report_scalar(title=”Title”, series=’validation’, value=my_metric, iteration=epoch_num)
“OK, I said I’d leave you alone. That’s it. I’m out.”
So you do this just to get the Trains evangelist off your back. Back to your model.
Some extra stuff comes up on the command line. Oh — Trains sees the git repository you’re using. Reminds you to commit your code. Then backs away. That’s kind of curious, it seems a little … extra. Imported modules don’t usually go sniffing around git. Hmm. It does have a point that you haven’t done a commit for two hours. Anyway, off your model goes.
Your model runs as usual. Another extra message from Trains appears, alerting you that your results are available via URL. Trains has spun up a test server for you.
You open the URL.
Whoa — it’s the same output as your command line. Trains has tunneled under your _print_ statements, in order to scoop up your terminal’s output. It’s still appearing in your command window as usual, it’s just being mirrored off to this web page as well.
You know how after a half dozen model runs, all that text gets lost in the window’s scroll or accidentally erased when you close it? Now you can always scroll back and never lose it.
Click.
OK, now tab over to the Scalars. There’s a graph of your validation metric updating in real time. Yes, I’ll wait while you add another one for your training metric to watch for over-fitting. Maybe some batch-level metrics too?
Sure, you could have written your own code to dump print statements to a file. And throw up a plot. But would you have? You are, after all, supposed to be doing 2 other things right now.
And this is just a couple of lines of Trains code. Anyway, it’s done now. (And Trains just reminded you again to check in your code, which you really should do.)
“So, what are you doing?”
Scooping up your output and making pretty graphs is enough reason to leave Trains in your working code. It’s nothing more than a web-based logger at this point, but it’s nice to have.
Maybe a couple days or weeks goes by. You’re busy. You’d like your friends and your mom to think you are busy thinking big thoughts, shaking the foundations of Artificial Intelligence, on the brink of incepting Ava from the movie Ex Machina.
The truth is, you’re fiddling with hyperparameters. Endlessly. The worse truth is, most of your time is spent this way. Every model, whether conventional or innovative, has a handful of knobs to turn. Any one knob can throw all the others out of whack. You can’t tell if a model has improved without checking and rechecking the entire knob-space.
Sometimes this is an exhaustive search; most of the time it is guided by intuition. And luck.
Trains hates to interrupt you again, but it can totally help with that.
“Let me drive.”.
Trains can drive all those hyperparameter tests, if you’ll just toss it the keys. This isn’t as dumb-simple as the initial client installation, for this you need to install a local Trains server on your development machine. If your experience is like mine, you’ll find the install so fast and fool-proof that you’ll gain the time back the first day you use it.
Just follow the directions here, Trains Server Installation.
Now when you run your model, the Trains URL will be to localhost on your development machine. One last thing — in order for Trains to tweak your hyperparameters, your code needs to accept them as command line arguments in the usual argparse way (the standard way you had meant to add but didn’t get around to it yet. Trains will make it worth your while.)
OK, so you’ve installed Trains and run your model. Trains brings up a URL hosted on your development machine. Only now, there’s a whole lot more to see.
Under Projects you’ll see your program. Click Clone. Now Hyperparameters.
Whoa! Yea, it knows all your hyperparameters. It’s got them assembled for you to edit in order to fire up another model run.
Now — you’re a coder. You pause with a : “Hold up. Did their code … just seriously go pull my parameters out of the argparse object so I wouldn’t have to specify them?” Sure seems that way.
Click.
At this point, something of a new world opens up as you can clone and queue runs for different hyperparameter settings with just a few clicks.
And it’s all being recorded — nothing is lost. Oh — and the logs too — and the graphics. It’s at this stage you begin to feel that Trains is reducing your workload, like some tireless intern at your side.
Another transition is also under way. Your attention is shifting. The desktop with your model-code is of less interest as your gaze increasingly drifts to the Allegro Trains desktop where you can drive, see results, go back in time. Sure, your model-code is still important. But only the dozen lines you’re thinking about. You’re not scanning up and down to tweak hyperparameters and all that. You’re no longer editing your code so much as thinking at it. Just the meaningful part. Trains has started to improve your focus.
This transition has only just begun, but it won’t be long at all until you iconify that command window entirely and just a critical chunk of code remains. Like a scaffold, the Trains environment wraps your model in so much functionality that it becomes the first place you go.
Chris Reiss is an independent researcher in Machine Learning, software developer and freelance writer. He is passionate about Open Source (he helped code the Chrome/Safari browsers!), occasionally gets picked up by major media outlets : (Business Insider, Fatherly, Life Hacker.) He does most of his writing on Quora, you can follow him here Chris Reiss on Quora .
