Can DevSecOps Undo DoD’s Broken Software Failures?

The Department of Defense (DoD) has been developing software intensive systems for the last thirty years. Only in the past decade has the Department openly recognized that these software intensive systems are critical to the future of U.S. National Security. DevSecOps, short for Development, Security, Operations, is one of the hottest commercial information technology (IT) trends. The DoD is now betting that DevSecOps will help the Military Services more rapidly deliver cutting edge Artificial Intelligence (AI) enabled applications to enhance combat operations.

So what is different about DevSecOps compared to traditional software development? Wikipedia defines DevOps as: “DevOps is a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality.” Because all software today must be deployed with some form of cybersecurity certification, DevOps has morphed to make cybersecurity a central component of all software development: “DevSecOps is an augmentation of DevOps to allow for security practices to be integrated into the DevOps approach.”

DoD DevSecOps

In practice DevSecOps (DSO) is a set of software development and cybersecurity tools used to continuously validate incrementally developed software added to a developing system baseline, generally on a bi-weekly or monthly iterative timeline. Using DSO tools and configuration control processes allows software development projects to integrate high quality software from many small teams, often geographically distributed, more rapidly and with continuing upgrades. In addition, DSO is usually set up to share reusable application code, often using shared-data, to enable the creation of new or enhanced capabilities for the intended users.

To date, DoD’s attempt to better integrate complex Systems-of-Systems (SoS) has been through Lead System Integrator (LSI) contracts set up to integrate previously independent systems into highly integrated warfare capabilities. This is primarily accomplished by sharing data, across software-controlled interfaces, to enable enhanced interdependent functions. More recently, partly as a result of Congressional push against failed industry led LSI efforts, Military Service System Centers have been leading LSI system integration efforts, but these have also resulted in long development timelines, cost overruns, and missed performance requirements.

The software component of complex SoS capabilities have now grown to become the dominant and controlling element of most DoD programs. This is evidenced by terms like software defined radio, big data analytics, autonomous flight control, artificial intelligence/machine learning, and cloud computing to include Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Infrastructure as Code (IaC). That complex systems have become dominated by software isn’t a surprise. It’s the natural result of computer processing growth that has continued to follow Moore’s Law over the last half century.

The first integrated chip was built in 1962. By 1965, Gordon Moore, Director of research and development at Fairchild Semiconductor, had already predicted, in a famous Electronics Magazine article, the annual doubling of component density (included transistors, resistors, diodes, or capacitors) on integrated chips. Although modern chips are reaching law-of-physics limits, Moore’s Law has continued to successfully predict the growth of chip density. Based on this, it is currently predicted that by the year 2025 a thousand-dollar computer chip will be able to process more cycles than a human brain and more cycles than the human race by 2045. This has come to be known as the singularity when machine growth becomes uncontrollable and may eventually dominate the human race.

IBM Q Quantum Computer

While classical computing continues its growth, revolutionary quantum computing technology is beginning to make progress by taking advantage of the strange properties of quantum mechanics. Today’s quantum computing is more about research than practical application, but has already given rise to Nevin’s Law of Quantum Computing. Neven’s Law predicts that quantum computing will “doubly double” processing power over conventional computers, and lead to quantum supremacy (the ability to perform a task no classical computer can achieve). In October 23rd, 2019, the scientific journal Nature, published an article detailing Google’s claims that, using their 54 bit quantum computer, they processed a task in 200 seconds that would take a classical computer 10,000 years. IBM refuted that claim by showing how restructuring the task, it could be done in two and a half days, thereby leaving the quest for quantum supremacy open. And on September 5th a Chinese physicist announced that they had a quantum computer, yet to be verified, that is one million times faster than Google’s quantum computer.

The necessary result of classical and quantum processing growth has been the growth in the size of computer programs, or software, because without software to direct compute cycles into needed functions, they are of no use. This has continued to challenge DoD’s LSI efforts that have, more often than not, failed to achieve the required SoS capabilities. More recently government led complex software LSI efforts have also failed to deliver needed capability on an acceptable cost, performance, and schedule, program plan. That brings us back to industry, and now DoD’s, use of DevSecOps tools that are intended to transform the way software is successfully developed. 

The good news is that the commercial growth in software tools, languages, and cloud computing, have radically changed the landscape for software development. The bad news is that during this same software technology growth period, cybersecurity threats have advanced significantly, elevating the need for cybersecurity hardened software in all areas of human endeavor. By bringing together advanced software development tools in combination with cybersecurity tools, DevSecOps has recently matured to become an ideal way to develop and sustain software intensive systems.

Because these modern software development and cybersecurity tools have reached a level of sophistication unavailable in past software development projects, one could consider automated DSO tools, coupled with best practice development processes, to be the equivalent of an automated software Lead System Integrator. Fortunately, this automation has significantly reduced the need for traditional human intensive LSI management functions. Said another way, a robust DSO software development process can be thought of as an Automated System Integrator (ASI), thereby, replacing much of the traditional human-intensive software development and documentation requirements with automation that enables rapid delivery of cybersecurity certified applications, and upgrades, in effective time sensitive cycles.

The net result is that DoD’s future cyber resilient warfare applications should be transitioned from government led LSI, year-to-multi-year development cycles, into quarterly ASI development cycles, delivering application and application upgrades, that are cybersecurity certified with operator validated capabilities. DSO may provide the pathway for today’s IT intensive systems to better pace the rate of change in modern warfare, and help ensure that Military operators are delivered the time-sensitive capabilities they need to prevail on the battlefield! 

This entry was posted in Uncategorized. Bookmark the permalink.

22 Responses to Can DevSecOps Undo DoD’s Broken Software Failures?

  1. Merv Cutler says:

    This article is very insightful and full of great information and background. Marv, you have done it again. Made us more knowledgeable by covering the topics we don’t always have time to involve ourselves with. Thanks. Great read!

  2. Marvin:
    Your recommendations are academic and generic and do not tell exactly what should be done, how and where.

    When you recommend that “… DoD’s future cyber resilient warfare applications should be transitioned from government led LSI, year-to-multi-year development cycles, into quarterly ASI development cycles, ..that are cybersecurity certified with operator validated capabilities,” ….you need to specify exactly what organization and what process must be changed – and how.

    You need to explain exactly what institutional process should be deleted and or replaced.

    Cordially, Paul A. Strassmann

    • Marv Langston says:

      Paul, You are correct. It was more about what is happening at a high level rather than being more specific at the lower level. Air Force is doing a pretty good job, from what I can tell from the outside,at pushing for a common DSO around their Platform One DSO/Cloud offering. Navy, at high levels, is trying to be common with them through the Navy partition they call Black Pearl. I believe that the DoD, at the DepSec level should mandate all DoD software be developed and sustained in a common DSO and multi-level cloud infrastructure… but not likely to happen:-)

  3. William A Schmitt says:

    Marv,

    A ray of hope in what I had otherwise seen as a cybersecurity death spiral. However on a different vector, I remain concerned that we are continuing to develop ever more complex systems of systems software that aggregates functionalities and control beyond capability of testing to verify all possible outcomes, thereby risking unintended consequences. But I am also no longer current in software development and testing methodology so hopefully my concerns are being mitigated. Your insight on same would be appreciated

  4. Elizabeth Dreicer says:

    Very insightful Marv. Thanks for sharing.

  5. Irv Reud says:

    As a software developer for 3 decades it appears that these thoughts are of the utmost importance for DOD to consider relative to moving faster. Very interesting.

    • Marv Langston says:

      Thanks Irv, your background gives your comments the credibility needed… good to see you joining the conversation…

  6. Rex Buddenberg says:

    Marv,

    One of the chronic problems in the military is the indefinition surrounding System of Systems. I’ve been away from this stuff for a while and this is the first I’ve seen the Wikipedia article. Better than nothing, but misses some critical points without which we don’t have a taxonomy.

    I. Information systems all have sense-decide-act cycles; critically with a communications system to connect the sense/decide/act nodes. The key is how we handle complexity; there are two ways:
    A. nesting where one sense-decide-act loop is nested inside another. Precision guided weapons and their host ship or aircraft are excellent examples.
    B. chaining where the output of one sense-decide-act loop feeds another(s). There are 1001 traps, both technical and programmatic that make chained systems difficult to bring in (multiple exercises for reader in here).

    The second omission (or undertreatment) of SoS is poor modularity. This often happens within a single program but is endemic when the SoS crosses program boundaries. Closely related t poor modularity is lack of growth headroom — requirements always change, and, as you’ve pointed out, available technology changes. Well-modularized systems nearly always have this headroom.
    Further, good modularity always exhibits data flowing from sensor to decider to actor via a clearly identified communications system (today and internetwork). Aha, this is the point where the data needs to be protected (always for authenticity and often for confidentiality).

  7. Refugio Delgado III says:

    Marv, nice article, we need the vendors providing the “DSO” to avoid or illuminate dependencies they “cook in” in order to differentiate themselves from all their competitors and also get our computer scientist to take advantage of parallel programing needed to take advantage of the new hardware and software operating systems being provided by industry. Thanks for “pushing” us all to think about what we are doing!

    • Marv Langston says:

      Thanks for pointing out some of the concerns hiding under the covers of this complexity… We must continue down the path of tools that make transparent the software that drives National Security…

  8. Michael Martorano says:

    Marv, the article opens up an area that provides a better method for validating software that complies with cyber security. Recently, ( actually over the past 5 Years) I have been developing an optical tracking system that observes missile and targets from launch, with the objective of supporting range safety decision to keep the object flying as long as it maintains a trajectory inside range safety boundaries. The software algorithms were challenging to be tested and certified by range safety but this was easy compared to my nightmares with getting cyber approvals ( interfacing with Navy networks)and also ATO( Authority To Operate), required approvals from Cyber command, then also DCSA. This required performing scans, and also a host of other testing verification which is time intensive. So the software development was challenging but satisfying the cyber elements ( artifacts) was more technically challenging. Lesson learned is that the software development should be integrated in some manner with cyber operations requirements earlier in the process, saving time, frustration, and money.

    • Marv Langston says:

      I completely agree Mike. One of the main purposes of the DoD DevSecOps efforts, like the USAF Platform One, is the ability to get an ATO as a byproduct of the DSO tools and processes, to include a continuous ATO for all upgrades and patches to operational systems. As you point out, this is a critical issue going forward.

  9. Junaid Islam says:

    Marv

    Your analysis correctly points out that DevSecOps is critical to national security. However to recruit/train/retain people with the desired skills DoD must do 3 things:

    1/ Allow time to develop expertize
    Real expertize only develops over decades of hardwork. Unless DoD can provide the envirorment where people can build their skills we’re not going to have the people you need to build a tactical quantum computer or hybrid space-ground C2 system or anything else that’s complex.

    2/ Provide a career path for software experts
    As the nature of warfighting changes so should the warfighter and their commander. Unless a coder can climb to the rank of General or Admiral you’re not going to have an effective command structure.

    3/ Offer competitive salaries for world class experts
    Perhaps the most difficutl issue for DoD is addressing the salary gap. Without a doubt, there are world class individuals at DoD who give up everything to serve. Unfortunately it’s a financial hardship that should be unnecessary. DoD must find a way to pay world class experts market rates or create some type of structure to enable them to monetize their skills yet serve.

    Junaid

    • Marv Langston says:

      Great points Junaid. And relative to salaries, if the DoD doesn’t quite contracting based on inputs like software engineer hourly rates, they will never be able to achieve effective outputs.

  10. Dick Rumpf says:

    Marv,
    I always learn so much from your Blogs. Keep up your good clear writing. This is not my field as you know so I really appreciate your sharing of good IT technology which significantly helps to educate engineers like me. I also enjoy the comments your Blogs elicit and your thoughts in addressing them. Be healthy and stay safe.

    Dick

    • Marv Langston says:

      Thanks Dick, good to hear that they are providing some value… No matter where it all goes, critical software will continue to drive our Military capabilities…

  11. Thanks Marv! Great overview of the situation. Lot’s of companies want to get involved in helping DoD do “DEVSECOPS”, but so far, I don’t see it as a solution that’s “sold” to DoD. DoD has it as a priority on almost every chart they show these days. Will be interesting to see how they achieve it. Agree it’s important.
    Personal example: I bought a state of the art Honda CRV with every conceivable bell and whistle in 2018. Off the floor, it was already old technology, and despite now two years of my driving it, the car hasn’t learned one single thing! I can’t buy a whole new car every year to get newer technology! I’ll go broke!

    • Marv Langston says:

      Thanks Chris, and that is why Tesla has decided cars could be done a different way… On the new Tesla Y, owners can decrease the 0-60 mph time by half a second for a $2K software update… and even just maintenance updates deliver software updates every 30-60 days…

  12. Eric V. Gruenloh says:

    Marv,

    Thought provoking as always.

    It sounds like many here would enthusiastically agree that the pace of technology is providing critical enablers for solving many USN fleet war fighter needs.
    However, I also sense a collective frustration at harnessing and achieving these benefits. I submit this entire blog warrants a symposium to exchange and/or vent ideas and limitations.

    To that end, in this environment, there is a critical construct outside of technology that must be addressed for the USN to realize these technology benefits. First let’s start with the following assertions:
    * Acquisition breeds Competition
    * Competition breeds Performance
    * Performance breeds Value-For-Money

    These are desirable acquisition attributes and many here would agree that your assertion of Quarterly ASI development cycles is technologically feasible. However, the reality of today’s acquisition environment means it is not unusual to see a Unilateral Funding Modification, on an existing contract, take 6 -8 weeks to process. Competitive tasking typically takes much longer to get started.

    Alternate contracting and multi-award approaches promised some relief. However, those benefits are diluted if they are shoehorned into an existing acquisition and financial accounting systems of one-size-fits-all checks and balances. These can literally take longer to complete than the desired software function takes to develop, integrate, and test.

    Additionally, standard system engineering processes and non-Title 10 oversight need to evolve and keep pace to control system development “entropy” and “requirements creep.” This is essential to ensure that the software funding applied to DSO efforts, produces the desired end-state functional and performance results, at the available higher productivity rates.

    So, IMHO the greater holistic challenge for this USN community continues to be how to absorb and realize the economic and tactical benefits of rapid technology advancement; whilst sustaining the performance growth and value-for-money benefits of competitive acquisition.

    “We don’t get to arbitrarily decide how good is good enough or how fast is fast enough. Our adversaries get to vote on that.”

    • Marv Langston says:

      As always, good points Eric. I think your slogan at the bottom says it all… We no longer have the luxury of waiting for inefficient acquisition to slow down improvements to our Navy… And in the Pacific it is the Navy that is front and center…

Leave a Reply

Your email address will not be published. Required fields are marked *