The Story of the BMLT
META: This is a copy of the case study on my Web site. It isn’t something that everyone will find interesting, but for some people, it’s absolutely fascinating.
QUICK META NOTE
This article contains a lot of technical jargon. That’s because I am giving a fairly detailed account of the decisions and considerations that went into the BMLT. You don’t need to understand the jargon (although I do provide links to fairly basic explanations of the topics, if you want to explore further). You can just glaze your eyes when the geekspeak starts, and skip over it.
WHAT IS THE BMLT?
“BMLT” stands for “Basic Meeting List Toolbox”. The system is a standardized way for Web sites to implement accurate, powerful meeting lists, and also affords semantic access, which allows introduction of derivative projects, like the BMLT iOS Apps.
It’s a large-scale project, run as a fairly classic open-source initiative, and has been in place since early 2009. In the time since, the project has grown to have a daily user base in the tens of thousands. It lists more than 29,000 regularly-scheduled weekly meetings all over the world, and has been localized into several languages (with more on the way).
The BMLT was designed specifically for NA meetings. It is a 100% volunteer effort (it’s free), and has been implemented in over 40 Servers, which, in turn, Serve over 1000 individual chapters of NA.
WHY WAS THE BMLT CREATED?
At the turn of the century, the worldwide Web was taking off. Everyone had to have a “Web page.” Hundreds of NA sites appeared; each, unique and insular. There was a desperate need for a unified, accurate and modern way for people to find NA meetings. People could not find meetings that gather just around the corner, and often were directed to nonexistent meetings.
NA is only one player in the addiction treatment field, but it is effective for those that choose to follow it. As is the case with most addiction treatment organizations, it saves lives. That means that NA meeting lists save lives. That’s not hyperbole. It was extremely important that the BMLT work, work well, and be widely adopted.
THE PROBLEM DOMAIN
The Talent
NA Service is completely voluntary. It’s considered an “obligation” for NA members to participate in NA Service, and no one is paid a dime. This means that the “labor pool” is extremely diverse, and you can’t coerce or demand things from its participants. You need to take what you get, and be grateful; also, accept that not everyone will be as motivated as you are (or may be motivated beyond all expectation). I strongly suspect that anyone working for a nonprofit organization knows exactly what I mean.
You can’t assume that there’s a lot of technical expertise. There’s often a great deal of passion and desire, but a lack of know-how. Just because someone has the position of “Webmaster,” doesn’t mean they have the level of skill and experience you’d expect in a professional Webmaster. They don’t usually have much of a job interview, besides raising their hand when someone says “Any volunteers?”
Another issue is that turnover in NA Service is extremely high. This is by design. As NA Service is an obligation, it is important for NA members to “get out of the way” for the “next generation.” A typical tenure for any NA Service position is one or two years.
The Resources
NA Service bodies run on a tradition of “enforced poverty.” They are, by fiat, supposed to be poor. If they really need money for something, they can fundraise, but they generally need to get by on a mere pittance and volunteer efforts. NA is also constrained in that all fundraising needs to happen within NA. NA Service bodies are not allowed to accept funds from non-NA members.
This means that you can’t expect their Web hosting options to be very robust. They are likely to have the cheapest LAMP server possible, running on a shared hosting plan.
The Landscape
First, NA doesn’t really keep track of its membership, but there is some information available (Downloads a PDF document). It’s quite large, with membership likely to be in the low millions.
NA has thousands of Service bodies, spread around the world. Each is generally focused on Serving its local constituency. NA World Services has a global bailiwick, but that’s quite unusual. NA Service bodies exist in almost every nation on Earth, and often have a character and resources that are quite dependent on the NA members that participate in each Service body.
NA is also quite decentralized (AKA “disorganized”). This is deliberate. It’s designed that way to make sure that local Service bodies concentrate their Service on their local needs. These Service bodies have minimal contact with other Service bodies; usually inversely proportional to distance. There are no “bosses” in NA, and even NA World Services can’t dictate to Service bodies or Groups. In fact, getting people on the same page is a perennial challenge in NA, and it would not be an exaggeration to say that it’s quite common to find NA members and even entire Service bodies, in direct opposition to any form of standardization or structure, if even suggested from anywhere else.
This means that having standards that apply to multiple Service bodies is very difficult. There are no standardized meeting lists, phonelines or worldwide assets, beyond those provided by NA World Services. Held in common is a set of “talking points” and philosophies that come from the NA literature curated by NA World Services. “Branding” is very loose, and even meeting formats and Service body policies vary wildly throughout the organization.
Another very important characteristic of NA is that it cannot allow itself to become “beholden” to any outside interest. That’s a big reason for the enforced poverty and refusal to accept outside sources of funding. It’s vital that any Service done in NA be as independent as possible of outside influences. Anonymity is also extremely important, so privacy and data security are at a premium.
The Need
Regardless of the patchwork of available talent and resources, NA is about saving lives. Its mission is quite critical, and the people it is trying to Serve are very often in the most dire straights imaginable. Their target audience is usually destitute, distracted, desperate and in despair. This means that Service aimed at them can’t be complex, time-consuming, bureaucratic or expensive.
The Summary
So that leaves us with the need to apply a global, organized, secure, simple-to-use and simple-to-implement, accurate meeting search engine to a large, disorganized, poor, polyglot organization; staffed by an unpredictable, ever-changing member base, and with a need to remain fiercely independent.
Quite a challenge.
We needed to design a system that would be cheap, secure, simple, with a flexible implementation on even the most basic servers. Additionally, the system needed to allow its implementors to be as independent and insular as possible.
Not only that, it needed to be written in a manner that encourages Service bodies to work together, but does NOT require cooperation –even with the authors of the system.
Carrot Vs. Stick
Enforcing or requiring support for the BMLT was a non-starter. It was not going to happen. Even though I worked in cooperation with NA World Services, I knew that it wouldn’t be something that could be forced or even strongly recommended to NA Service bodies. We needed to make sure there was plenty of “carrot,” and virtually no “stick” at all. People had to want to use it; not have to use it.
THE PLAN
The start of the BMLT was a philosophy:
“Make it easy to install, easy to use, super high-quality, free, and reward everyone like it’s perpetual Christmas.”
Pretty straightforward. I needed to make sure that we had “easy” and “reward” in the basic DNA of the BMLT. That informed everything else I did.
Quality Is Job One, Two, Three, Four…
Of utmost importance was quality. This was going to be a platform, not an app. I knew from the git-go that this was going to be a major initiative, probably years in development and refinement. If it was to be successful, then it would be used by hundreds of thousands, if not millions, of people.
And I knew that I would probably get very limited help, if at all. I was destined to be Head Cook and Bottle-Washer; possibly for the entire life of the project (NOTE: This is not by choice. I’m a control freak, but I’m not that bad).
So the quality needed to be “through the roof.” It needed to be some of the finest software I’d ever written, because problems would be multiplied a thousandfold, and, if I was alone on the project, I wouldn’t be able to handle it. This was a “nights and weekends” project, with the scale and scope of a major-league, commercial project.
Keep It Simple
The operation, installation, use and even the code, itself, needed to be as simple as possible.
Because of the variety of talent implementing the BMLT, I couldn’t expect people to be expert geeks. I needed to ensure that installing and maintaining a BMLT Server would be painless and straightforward.
I needed to make sure that the end users would have a highly usable (read: “simple”) experience. It was very important that they be able to rapidly find meetings without having to jump through too many hoops.
I wanted to be able to invite people to get involved, extend and maintain the project, as well as be able to audit it. This meant that the code needed to be basic and simple.
I’m a fairly experienced and advanced programmer. I know a lot of good “tricks” and patterns. Some are very clever and can have amazing results.
And I couldn’t use them in the BMLT, because I needed to make sure that even fairly inexperienced and casual developers could understand it.
Meeting Frequency
I decided that the meeting search system would be for normal NA Recovery meetings that gather regularly, once per week (or more often), at the same time, in the same location. This is the typical pattern for NA meetings.
Over the years, I have gotten requests to have “special” meetings, such as Service committee meetings, or ones that only meet on certain special occasions, added. I have always resisted this, because we need to do one thing, and do that one thing well. It’s a bad idea to degrade the entire system for 99% of the users, in order to improve it for 1% of the users.
Location, Location, Location
I wanted the location to be a stored long/lat pair; not something dynamically calculated from the address (like most of these types of applications).
The reasons for this were twofold:
- Oftentimes, geocoders (the formulas that calculate long/lat from addresses) are inaccurate. Sometimes, by a lot. This can be the fault of the people that entered the address; not the geocoder. Remember what I wrote about “rewarding” up there? Part of it was giving the administrators as much flexibility as possible. If they want to enter junk for an address, then let them. The long/lat can be set by actually dragging the marker in the administration map, so the marker/long/lat location doesn’t have to be related at all to the text. Also, you can sometimes have meetings that gather in places such as parks or beaches, where there’s no way to generate a geocode from the address.
- The other reason is that having the long/lat embedded in the database allows for much faster location-based searches. I wanted to embed a haversine formula into the database lookups, so that I could do a very fast “find meetings near me” search. One of the most integral parts of the BMLT meeting search experience is the “Auto-Radius” location search. The idea for auto-radius actually predated the initial envisioning of the BMLT, but it was so “natural,” that it became central to the system. This needs to have long/lat embedded in the database, not calculated dynamically (Yes, you can do it dynamically -if you have a monster server. That’s a non-starter for NA. We need to work with the cheapest shared accounts possible).
Over the years, I have been asked to add phone-based and Web-based meetings to the BMLT. Like the “special” meetings, I have resisted these requests, because long/lat location is central to the way that the BMLT works.
A Declaration of Independence
It was quite important that the BMLT be developed as a standalone, independent project. This was mostly a “political” measure, as there are many different orthodoxies and philosophies within NA, and attachment to any one of them would “poison the well.”
That said, it was important to work closely with NA World Services, as they were the closest you could get to a “central authority” in NA, and they also worked with new, or developing, areas.
A BMLT Root Server can output data that will help NA World Services to keep their meeting list up to date.
Support and Documentation
A full-featured support Web site would need to be created. This has been a major project since its inception, and is still very much a growing entity.
Additionally, it was important that the project be 100% open-source, with direct access to all source code repos.
“Crowdsourcing”
A very important aspect of the BMLT, was that I wanted the meeting lists to be maintained by the people that actually attended the meetings, as opposed to central administrators. This would ensure that the meeting information was as accurate (and locally-relevant) as possible. It would also make big jobs a lot smaller. If you can delegate portions of a big task, it makes life easier all around.
Another benefit of this model, is that the more people that are involved in administering the system, the more “trained up” people you have. You have a significant cadre of folks that have 90% of the training they’ll need for the “next level,” already in place.
At the time that the BMLT was conceived, the term “crowdsourcing” wasn’t en vogue, but that was just what we meant; whether or not a term had been invented for the concept (which wasn’t exactly new, anyway).
The security and database models were formed around the concept of delegating administration. This was a bit of a departure from “traditional” NA meeting lists, which tended to be maintained centrally. I didn’t want to make it difficult to do central maintenance (I was quite sure that many Service bodies would still want to do central administration), but I wanted to make it easier for more local Service bodies to do their own meeting list administration.
CHOOSING THE TECHNOLOGIES
Why Did We Use PHP?
I decided to use the PHP Programming Language for the server scripting.
There were a number of reasons for this. The biggest one, was that PHP was becoming the de facto server language of choice for people writing free software. It was NOT the best, most robust or secure language available for the server, but it was enormously popular.
In 2008 (when the design for the BMLT was being developed) many servers were still running PHP version 4, which was rather primitive, in many aspects (Some would argue that it still is, but it’s come a loooong way). This meant that I had to reduce the amount of Object-Oriented Programming in the system, as PHP’s object model was pretty bad, and many PHP programmers were not able to understand Object Programming.
By the time the BMLT was released, I was able to require PHP version 5 and over, but it wasn’t really in place at the time that the design was being done.
So that’s why there’s so little object code in the system. I restricted the object code to the Model layer, which was the internal “engine” for the BMLT. This was code that would almost never get changed, and no one would need to get at.
Why Did We Use MySQL?
To be honest, I did not decide to be MySQL-only at first. My goal was to be “database-agnostic.” I used a PHP utility called PDO to access the database, using PDO Prepared Statements. This is an excellent security practice. When you use PDO Prepared Statements, you implement a very low-level defense against SQL Injection Attacks, one of the most common hacks in the world.
PDO is also, on paper, “database-agnostic.” You should be able to run it over several different types of SQL servers.
In practice, it isn’t quite that simple. There’s a lot of fairly fundamental differences between the various databases, and I found that I had to pretty much stay with MySQL. It would have been a prohibitive amount of work to write (and test) support for multiple databases.
Which was good, because MySQL commands an enormous swath of the cheapest hosting solutions available.
Lighting the LAMP
“LAMP” is an acronym (geeks LOVE acronyms) for “Linux Apache MySQL PHP”. It describes a standardized Web server package that runs on the Linux server operating system, running the Apache Web Server as the software that actually serves up Web pages, the MySQL Database Engine to store data, and the PHP Programming Language as the server-side scripting language.
LAMP is still a massively popular server stack. In particular, LAMP still rules the “budget-conscious” Web server market.
That’s where you’ll find most NA Web sites.
Additionally, because of the past prevalence of LAMP, the new servers that are starting to become more popular generally stick to the same protocols as LAMP, so it’s a good idea to write your Web software to support LAMP. It means it will probably run fine on many other types of servers (the BMLT is currently deployed on many types of servers; not just LAMP).
So that’s why the BMLT uses PHP and MySQL, and the code seems so simple. LAMP is a fundamental, basic server stack, the code and the database schema needed to be simple enough that it wouldn’t take a Fulbright Scholar to decipher it, and the demands on the software wouldn’t be that high (even if every meeting in NA were being tracked by the BMLT, it would barely make the bandwidth meter twitch, compared to the monster systems in use by commercial and government entities, these days).
So that’s why I chose the particular technologies used by the BMLT.
Client/Server
I decided that the BMLT should run on a client/server model, which means that there’s a singular central server (We call it the “Root Server”), containing all the meeting data, and multiple clients (We call them “satellites”) that connect to the central server in order to request and fetch data. This allows easy administration (one place to log in and manage the data), and lots of choices for fetching and rendering the data.
These satellites can be WordPress plugins or Drupal modules, as well as a basic (standalone) client, or even iOS apps.
The Root Server, itself, is a standalone Web application. It is installed by itself, and is invoked by satellites, which may (or may not) be attached to some other software system (like WordPress).
Security
I decided that security was something that needed to be designed in since Day One. It was necessary to have multiple administrators, with varying levels of access, and the ability to divide meetings into domains, tended by administrators.
This meant IDs and passwords, as well as some kind of login persistence. It was important that we be able to isolate the meetings available to certain admins, and that the system be easy to use.
As any programmer knows, “Secure” and “Easy” don’t usually live under the same roof.
Measured Response
Most security vendors try to tell you that you need Fort Knox security for even the most innocuous use cases. This results in things like employees of an auto repair shop being unable to visit a parts supplier’s Web site from on-premises computers (Don’t laugh. I’ve seen much worse).
You definitely should have that kind of security for some purposes, like banking, IT security, medical records, defense stuff, etc., but you shouldn’t have air-gapped computers in a soup kitchen.
I decided that the level of information wasn’t so sensitive that it required DEFCON 1 measures. There’s definitely a minimal level of security that ALL computers should have, simply because ‘bots don’t really care if your site is connected to your bank, or to the light switches.
I decided that persistent cookie wasn’t quite secure enough for me, as I wanted to be able to add semantic administration in the future, using the same login system, but that forcing people to log in repeatedly was a non-starter. For one thing, it actually damages security, as the browsers are constantly sending credentials. I don’t like that. I want the credentials to be sent once, then no more authentication information is to be sent.
Session-Based Logins
So I decided to use session-based login. It’s fairly convenient, as the user can maintain their administrative login while the browser is open, but will need to log in again after they close the browser (which ends the session).
Don’t Take My Word for It
Another security measure that I employ is pretty geeky and low-level. Whenever ANYTHING is about to be written to the database, the software, at an extremely low level, throws out all the user’s permissions (the ones the user says they have), and reloads the user’s permissions from the database, using the session credentials. This prevents the user from “escalating their privileges.” It’s a pretty simple way to spike a fairly sophisticated hacking method (of course, if they get into the database, this won’t do any good, but if they’re already in there, you’re screwed anyway).
Sequestration
One of the most fundamental security practices is sequestration. This means that users are only cleared to see certain information, and they are blocked from seeing anything outside of that. It’s a very, very old practice (I suspect hundreds -if not thousands- of years old). Most good security practices are old-fashioned horse sense.
The BMLT data is arranged in a hierarchical fashion; much like NA Service. Meetings are assigned a “parent” Service body. Each Service body is assigned to users that can:
- See meeting information (including some fields that can be hidden from the regular public), but not modify it,
- Modify meeting information, and
- Modify meeting information and Service body information.
Service bodies can be “nested” inside of other Service bodies, and permissions for the “parent” propagate to the “children.”
So this means that you can have multiple administrators; each, with their own login, that have specifically-assigned permissions, and specific groups of meetings and Service bodies they can edit or observe. It’s quite possible to have an admin that can edit meetings in one Service body, but be able to edit the Service body information, as well as meetings, in another Service body, while not allowed any rights in yet another Service body.
God Mode
I decided that there would only be one, single “God” administrator. This is an admin that can create users and Service bodies, and edit some of the more “stable” data, like format codes. It would not be an “Administrator” level of permissions. It would be a single user, with an ID of one. There are a number of operations that don’t look for permission levels. They look to see which user is doing it.
This results in acceptable security for what we have, and relative ease-of-use.
Just-In-Time Specification
One of the ways that I work, is that I try to “defer” as much specification as possible, because the use cases and environment of any software system (especially one as complex as I knew the BMLT would be) are chaotic and unpredictable. It’s asking for misery to try to plan the entire system lifecycle up front.
I decided that the BMLT would have two (at least) distinct iterations in its lifecycle: The first would be very server-based, where the server did all the thinking, and the second would be a true client/server model, where the clients were much more independent of the server.
The reason for this odd decision, was because I really wasn’t sure exactly how the BMLT would be used, and I also knew that various PHP-based content-management (CMS) systems were battling it out, and I wanted to see who would “come out on top,” as that would inform the API that I would develop to handle communications between the clients and the Root Server.
Version 1 of the BMLT was designed so that the actual HTML that was shown to the user of the satellite was created on the Root Server, and sent across the connection to the user. This is clumsy, at best, but allowed me to have extremely simple satellites that didn’t need to be too “smart.”
Version 2 would have a semantic API, and the clients would simply ask the server for data, and take care of interpreting the data, composing the user experience, and presenting it to the user on their end; a much better system.
The thing was, I wasn’t comfortable designing that API while I was working on the server. I wanted to put it off, until I got a chance to figure out where the bare spots were, and pave them over.
Now, almost ten years later, I believe that this was exactly the correct approach. I learned many things about how people used the BMLT, and also watched WordPress and Drupal come out on top of the “CMS war.”
Here is the current BMLT Semantic API. All of the BMLT satellites use it to extract data from the Root Server.
Localization
An important aspect of the BMLT, is that it’s designed to be used for many (non-English) languages and localities.
At design time, I decided to use token-based localization. This is a classic pattern, where no strings are directly written to the page. Instead, a function is called with a token (a placeholder string). The function loads a text file that represents strings for the current language/locality, and replaces the token with the proper string before returning the replacement from the function.
I also made sure that we could handle different time displays (Military v.s Ante Meridian), and weeks that start on days other than Sunday.
It was also important that all displayed pages use UTF-8 text representation, so that non-Roman characters (such as diacriticals) would be rendered correctly.
THE PROJECT
Initial work on the BMLT took almost a year until it was finally released in June of 2009.
Most of that time was spent on the “backend” (administration) code. As I mentioned earlier, the Root Server was building the displays for meeting searches, and sending full HTML page fragments to the satellites. This took surprisingly little time.
However, I wanted to make sure that the administration was effective and secure. I designed a fairly basic, but flexible database schema. This has proven so effective, that it hasn’t needed to be changed in eight years.
Release 1 (Server-Centric)
The BMLT was officially released on June 25, 2009. The first implementation for it was the Greater New York Region and the Eastern New York Region, which had about 1000 meetings, between them. Since then, it has expanded to cover over 600 Service bodies (including all five Regions of New York), and 20,000 meetings.
Release 2 (True Client-Server)
In April of 2013, Version 2.0 was released. This implemented a massive change to the Root Server, with a much improved administration capability and a semantic interface. It was followed soon after by new satellites that worked independently of the Root Server.
The Standard BMLT Satellite
One of the things that needed to happen with the BMLT, was that it had to be “end-to-end” complete. We couldn’t just create a server, and insist that users write their own satellites (although several people have done exactly that). In order to meet this requirement, I had to support the two major CMS packages (initially, there were three CMS packages, but I dropped support for Joomla), as well as have a version of the satellite that would work independently of any content management system.
I addressed this by writing the BMLT Satellite Base Class. This is a PHP package that encapsulates about 90% of the code needed for a powerful BMLT satellite. It was fairly straightforward to integrate this into a plugin (except for Joomla, which was one reason that I dropped support for Joomla). This was how I was able to provide a very capable satellite for the end user. It makes implementing the BMLT quite painless.
Additionally, the satellites are designed to be extended and modified, using stylesheets. They use very low-specificity CSS.
OTHER PROJECTS
Introduction of the Semantic Interface opened up the system to allow “third-party” derivative projects. Most significantly:
- The BMLT iOS Apps, which implement iPhone and iPad search capabilities. We are currently in the second generation of these apps, which are now worldwide, and also include an administration app. These are now driven by an encapsulated library that abstracts all communication with the Root Server.
- The Tabbed UI WordPress Plugin, which has proven itself to be enormously popular. It is a JQuery-driven tabular display plugin with powerful sorting and filtering capabilities.
- The Sandwich Project, a “Root Server aggregator.” It takes all the known Root Server instances, and presents them as if they were one, huge worldwide Root Server. It’s awesome.
- The Yap Project. This is a “text-to-speech” implementation of the BMLT that allows it to be integrated into phonelines.
- The BMLT Meeting List Generator Project. This is a site that allows Service bodies to create “on the fly” PDF meeting lists, suitable for printing.
IN SUMMARY
I consider the project to be a great success. Its lifecycle was quite unusual, compared to most projects of its type, but its organic growth was designed in from the start.
As of the date of this article, the BMLT has been localized in Spanish, Swedish, Italian, German and French, with partial translations in Danish and Brazilian Portuguese.
The BMLT has spread around the world, with all of Australia, India, Germany, Austria, Italy, Ireland, Sweden, almost the entire East Coast of the US, parts of Canada, Southern California, and a significant swath of Central and Southeast USA covered. More implementations are under way
You can see its influence, “live,” here. Once the tally has completed, and the table is displayed, select the “Display Coverage Map” link at the top (The meetings off the coast of Africa and in Antarctica are data entry mistakes by meeting list administrators).
