keeping it all together

[EspressoLover]

Disagree with others. DVCS is a game changer. The only real upside to the centralized approach is better handling of binary assets. But those should live in artifact repositories, not source control, anyway. When you move to git, what you realize is that commits and branches are super light-weight and local. With svn the tendency is only to commit "version changes" with sprawling footprints. With git you tend towards a separate commit every time you change a few lines of code. This just isn't feasible with svn, because each commit changes the repo for everyone else. With git, it doesn't matter if you break the build, because it's only local until you push.



Having much finer granularity on commit history enables all kinds of productivity boosts. git revert essentially becomes Ctrl-Z in your local workspace. You can "git log -p | grep" to effortlessly to exactly when, where and why some change was made. git bisect is literally a one-button solution to diagnosing bugs. Not to mention source control functionality stays completely available even when you don't have an internet connection.



Same story with branching. With svn, branches are a pain in the ass, and you probably only use them for major version changes. With git, I'll use local branches as ways to isolate even the smallest change sets. Let's say I'm working on adding some feature X to the codebase, when I notice some orthogonal refactoring Y that I want to do. Using git branch, you can easily toggle back and forth between each change, keeping the workspaces single-focused and the changelog isolated. If you're collaborating with Alice on X, you can push to that branch to her, then collaborate with Bob on Y. Neither has to worry about the tasks outside their purview.



Specific to your question, it also makes lightweight experimentation simple. You can fork a "skunk works" repo to add some experimental features. Keep it as long-lived as you like, merge downstream changes from master as needed, and selectively promote changes back into master. You can keep master hooked into a CI pipeline, so that you don't have to worry about untested experimental fork changes accidentally polluting the stable codebase.

[Maggette]

This might be considererd crazy...but for my "side projects" (a smaller project I spend probably one day a week on and private stuff) I use scrum tools (picture me running for cover).



I had the same problems with these side projects first. Work was started but often didn't lead to anything. I lost track of stuff..etc



When I first encountered Scrum I was a hater, and to a certain extend I still am.



But I used some parts of it for these side projects. It obvious to me, that I get more done, even though you "waste" some of your time on planning, writing and estimating stories (or tasks) and refinining stories. There is a trade of here of course.



But I do have a plan now. My forecasts about what I get done the next three weeks are shockingly accurate (and often depressing). But in the end I get more things done. The combination of jira ( Iuse https://www.openproject.org/de/jira-alternative/ ) and GIT are also a kind of a poor mans documentation.



I think it is not important that you use Scrum or whatever. But spending time on planing and creating and structuring tasks really helps.



Of course it's research and not a web service application. But it is totally ok to write down after a task => nothing interesting came out of it => no follow up activities.



To me it just feels by structuring it and doing it consciously you get much more out of dead ends.

[EspressoLover]

On the topic of organizing the research process, I'd say there's two major separate challenges. One, is code stability and maintainability, while keeping experimentation low overhead. Two, is data provenance.



Code



From a software engineering perspective, research is fairly challenging. The vast majority of code produced in a research context gets thrown away or never used again. Most regular software is produced against a relatively fix spec. I.e. there's a product spec that calls for X. When we write code to do X, it's likely it or some future version of it will stick around for the life of the product. Against that it's justifiable to keep strict requirements in terms of software quality. Test coverage, coding standards, documentation, maintainability, code review, backwards compatibility, etc.



This isn't really what you want for research. The median line of research code gets written once, used a couple times in the same environment by the same person who wrote it, then forgotten about in a few days. Stability is a much lower priority than making it easy for researchers to quickly experiment with ad-hoc solutions without a lot of formal overhead. Then there's another twist, in that some unpredictable subset of research code will eventually be promoted into production systems.



Dealing with this isn't simple. It's easy to get lazy and avoid good software engineering standards by pretending that core production code is still research. Vice versa, once you've been burned you may go overboard with formal requirements effectively shutting down innovation. Plus keep in mind that in most orgs there's a power struggle between researchers and engineers. At the end of the day it takes honest actors with good judgement to decide when, where and how to vary the standards between different parts of the codebase.



One thing that does help is at least being explicit about it. Keep written standards for different levels of code, with everyone on the same page, and be clear about when code graduates from one level to another. Alice shouldn't have the gut feeling that this is still informal experimental code, while Bob is shipping it in a critical system.



I prefer to keep the division simple, two levels: "skunk works", which is the wild west, and "core", which should always be production safe. Obviously only core should ever be called by core. But even once something's starting to get used across different places in skunk works it should get promoted. Once a sub-project is used outside a single research team, or revisited outside it's original sprint, or grows past a few thousand LoC, or splitting into multiple layers of abstraction, then it should probably be promoted.



YMMV. Depending on your use cases and org's personality a different approach is justifiable. Maybe more granularity than just two levels, or different guidelines around promotion, or some other variation. I don't think the details are as important as just articulating a coherent philosophy that you can justify.



Data



Most research is just churning out all sorts of intermediate datasets and derived parameters. Some of which get used further downstream to make more datasets and parameters. Some are getting pushed right into production. Some are getting put in front of a human researcher who's trying glean an insight or make a decision. It's often not really clear what the end-goal is when you're actually generating the data what it will wind up being used for. The way the data's generated depends on all kinds of subtle structure and logic.



Data isn't like code. It's not self-explanatory. It's just a blob of bytes, and how exactly we created those bytes is not inherently represented inside the data itself. The challenge is to keep a provenance of how the data was generated and what it actually represents. I.e. metadata. Making metadata useful is really tough, especially when the data comes from complex transformations. Metadata could potentially be much higher, dimensional than the underlying data itself. For example fitted alpha coefficients, you might have to track all the parameters used in cleaning and pre-processing, the version of libsvm used, the random seed used in the fit, the date range, the symbol set, all kinds of hyper parameters, etc.



The less data you need to provenance the better. Ideally the only canon would be ground truth data (e.g. raw capture logs), and code that's sub specie aeterni. Derivations of data (including parameters) are done on the fly as needed, and discarded after they're finished. That doesn't mean that you can't keep derived data cached, but it's treated as scratch work, rather than a canonical source of truth. As soon as there's any question about where it came from (e.g. was this made with the most recent version of the library), you just discard and regenerate, rather than trying to investigate the origins of the current dataset.



There may be certain barriers why this doesn't work. One is computational restraints. If it costs $500,000 in computer time and takes three weeks to fit some parameters, then generating on the fly won't work for you. Another is if the one-button regeneration isn't practical. Maybe at some point in the pipe a human actually needs to use their judgement to make a decision. Or your current software doesn't have the hooks for it (although something like Apache Airflow should make this easy, even if you're wrapping a bunch of disparate clunky systems). And most of the time actual production parameters should be stable, and not reset because of some minor commit in the fit library.



Even then, it's still helpful to focus on making the surface area for provenance small. If you're compute constrained, only canonize the most upstream transformation that's past the compute barrier. Anything downstream derive on the fly when it's cheap. The fewer artifacts that need metadata, the simpler schema you can use. Millions of of artifacts in canon are going to require a machine readable schema with every possible dimensional included to be on the safe side. But for a single production param set, the metadata can just be a plain English changelog. The strategist can just use her personal judgement about when a refresh is needed.

[Strange]

@jslade github provides a service. You can also provide that service for yourself if you want to waste time configuring things; I do it because I am paranoid.



Are there any secure services out there? I.e. someone that provides a DVCS, but has no access to your codebase? As a minor point of paranoia, I do recall reading that some fed agency has requested source code for some company and got it directly from Bitbucket (or someone like that).

[Strange]

@EspressoLover Lot's to think about. I was not thinking of changing my process, but now I might. A complicating addition is the fundamental/discretionary research notes and such.

[Rashomon]

great points as usual, EspressoLover. Rmd’s / jupyter / lhs help ith data provenance. Stitching these together across a large team is about as well organized as lawyers e-mailing each other Word doc revisions, though.



Strange: Yes, Amazon and Bitbucket/Github/competitors all have secure cloud storage for government / hppa / etc. The instructions on how to et up a git server yourself are pretty simple though, in schacon’s pro-git book. (It’s one chapter, you can do it in half a day.) I do it not because I’m paranoid, but because I hate SV. I wouldn’t pay for FiresideChat when irc servers are free and, given claims I would make in any interview about my level of technological ability.

[Strange]

@Rashamon I understand that running a git server is not a rocket science, but it all adds up time-wise. E.g. if I run my own git server, my own DB instance, optimize my own servers etc two things would happen - firstly, I'll do it worse than someone who understands how to do it well and I'll also waste time that I should be spending searching for that elusive alpha. Outsourcing non-core stuff makes a lot of sense, I think.



One thing I realized reading the above ideas is that I should stop emailing myself and switch to something else.

[ronin]

Meh. Keeping it all together is overrated. I don't think I have kept it all together since at least 1996.