Showing posts with label Java (General). Show all posts
Showing posts with label Java (General). Show all posts

Monday, September 27, 2021

More Frequent Java Long-Term Releases

A little over four years ago, Mark Reinhold (Chief Architect of the Java Platform Group at Oracle) stated in his blog post "Moving Java Forward Faster": "For Java to remain competitive it must not just continue to move forward — it must move forward faster." In that post, Reinhold proposed "that after Java 9 we adopt a strict, time-based model with a new feature release every six months, update releases every quarter, and a long-term support release every three years." Reinhold stated that the motivation for this was that Java "must move forward faster ... for Java to remain competitive." Recently, a little over four years since the "Moving Java Forward Faster" post, Reinhold has posted "Moving Java Forward Even Faster."

Mark Reinhold's proposal in "Moving Java Forward Even Faster" is to "ship an LTS release every two years" instead of shipping a long-term support (LTS) release every three years as has been done since the 2017 proposal was implemented. Reinhold adds, "This change will give both enterprises, and their developers, more opportunities to move forward" and "will also increase the attractiveness of the non-LTS feature releases."

In the "Moving Java Forward Even Faster" post, Reinhold explains the motivation for reducing the time between LTS releases from three years to two years: "Developers are excited about the new features — which is great! Many are frustrated, however, that they cannot use them right away since their employers are only willing to deploy applications on LTS releases, which only ship once every three years."

Reinhold posted a similar message to the OpenJDK general discussion mailing list. In that message, he points out that "the LTS release following JDK 17 would thus be JDK 21 (in 2023), rather than JDK 23 (in 2024)." Reinhold also points out, "This change would, if accepted, have no effect on the main-line feature releases developed in the JDK Project. Every such release is intended to be stable and ready for production use, whether it's an LTS release or not."

Reinhold concludes his message with a request for feedback via comments and questions. There have already been some interesting responses:

  • Volker Simonis has replied on behalf of the Amazon Corretto team: "I'd like to express our support for the new JDK LTS release cadence proposal. We think this is the right step forward to keep the OpenJDK project vibrant and relevant with benefits for both developers and enterprises."
  • Martijn Verburg has replied on behalf of the Microsoft Java Engineering Team: "We would also like to endorse the 2-year LTS proposal for builds of OpenJDK. Since most of the end-user ecosystem prefers to have the extra stability of an LTS, this is a great way to encourage them with their modernization efforts!"
  • Gil Tene has replied on behalf of Azul: "I'd like to express our strong support for this proposal to move to a more frequent LTS cadence in OpenJDK."
    • Tene adds, "It does come with an extra maintenance burden for the community as a whole, but that burden is well worth it in our opinion."
  • Andrew Hale of RedHat replied: "So, from me it's a rather nervous yes, but." Hale discussed why the decision was "nuanced" and depends on the perspective taken:
    • "As developers of Java as well as users of it, we're perhaps biased in favour of new features, and we're less likely to feel the adverse effects of the upgrade treadmill."
    • "From my point of view as an engineer, moving to a two-year LTS cycle is broadly positive."
    • "Certifying libraries to run on a new Java release can take months of effort, and no-one is going to welcome having to do so more frequently."
    • "Many of our end users have been highly resistant to upgrading to new Java releases."
  • RĂ©mi Forax has replied: "I've discussed with quite a lot of people during this week, most of them prefer a 2 year schedule for a LTS. 2 years is good for application developers, 3 years is better for library maintainers. There is at least 10x more application developers."

The Oracle Java SE Support Roadmap has been updated to reflect the two-year candence for LTS releases. It now states (I added the emphasis), "Oracle will designate only certain releases as Long-Term-Support (LTS) releases. Java SE 7, 8, 11 and 17 are LTS releases. Oracle intends to make future LTS releases every two years meaning the next planned LTS release is Java 21 in September 2023."

As a developer who uses the latest JDK releases for personal projects (and often even "plays" with the OpenJDK Early-Access Builds) and then deals with the frustration of using older versions of the JDK on different projects in my "day jobs," I believe that reducing the wait between LTS releases from three years to two years will be welcome.

Related Resources

Wednesday, September 15, 2021

The Case of the Missing JEPs

The JDK Enhancement-Proposal (JEP) process is "for collecting, reviewing, sorting, and recording the results of proposals for enhancements to the JDK and for related efforts, such as process and infrastructure improvements." JEP 0 is the "JEP Index" of "all JDK Enhancement Proposals, known as JEPs." This post provides a brief overview of current JDK Enhancement Proposals and discusses the surprisingly mysterious disappearance of two JEPs (JEP 187 and JEP 145).

JDK Enhancement Proposal Overview

The JEPs in the JEP Index with single-digit numbers are "Process" type JEPs and are currently:

The JEPs in the JEP Index with two-digit numbers are "Informational" type JEPs are are currently:

The remainder of the listed JEPs (with three-digit numbers) in the JEP Index are "Feature" type JEPs and currently range in number from JEP-101 ("Generalized Target-Type Inference") through JEP 418 ("Internet-Address Resolution SPI") (new candidate JEP as of this month [September 2021]).

Finally, there are some JEPs that do not yet have JEP numbers and which are shown in the under the heading "Draft and submitted JEPs" The JEPs in this state do not yet have their own JEP numbers, but instead are listed with a number in the JDK Bug System (JBS).

Originally, a JEP could exist in one of several different "JEP 1 Process States":

  • Draft
  • Posted
  • Submitted
  • Candidate
  • Funded
  • Completed
  • Withdrawn
  • Rejected
  • Active

The explanation of evolved potential JEP states is described in "JEP draft: JEP 2.0, draft 2." This document has a "Workflow" section that states that the "revised JEP Process has the following states and transitions for Feature and Infrastructure JEPs" and shows a useful graphic of these workflows. This document also describes the states of a Feature JEP:

  • Draft
  • Submitted
  • Candidate
  • Proposed to Target
  • Targeted
  • Integrated
  • Complete
  • Closed/Delivered
  • Closed/Rejected
  • Proposed to Drop

Neither these documented states for Feature JEPs nor the additional text that describes these state transitions describes a JEP with a JEP number (rather than a JBS number) being completely removed and this is what makes the disappearance of JEP 187 ("Serialization 2.0") and JEP 145 ("Cache Compiled Code") unexpected.

 

The Disappearance of JEP 187 ("Serialization 2.0")

JEP 187 is not listed in the JEP Index, but we have the following evidence that it did exist at one time:

It's surprisingly difficult to find any explanation for what happened to JEP 187. Unlike fellow serialization-related JEP 154 ("Remove Serialization") which has been moved to status "Closed / Withdrawn", JEP 187 appears to have been removed completely rather than being present with a "Closed / Withdrawn" status or "Closed / Rejected" status. Adding to the suspicious circumstances surrounding JEP 187, two requests on OpenJDK mailing lists regarding the state of this JEP (14 December 2014 on core-libs-dev and 6 September 2021 on jdk-dev) have so far gone unanswered.

The reasons for the complete disappearance of JEP 187 can be insuated from reading the "exploratory document" titled "Towards Better Serialization" (June 2019). I also previously touched on this in my post "JDK 11: Beginning of the End for Java Serialization?"

 

The Disapperance of JEP 145 ("Cache Compiled Code")

Like JEP 187, JEP-145 is not listed in the JEP Index, but there is evidence that it did exist at one time:

Also similarly to JEP 187, it is surprisingly difficult to find explanations for the removal of JEP 145. There is a StackOverflow question about its fate, but the responses are mostly speculative (but possible).

The most prevalent speculation regarding the disappearance of JEP 145 is that it is not needed due to Ahead-of-Time (AOT) compilation.

 

Conclusion

It seems that JEP 187 ("Serialization 2.0") and JEP 145 ("Cache Compiled Code") have both been rendered obsolete by changing developments, but it is surprising that they've vanished completely from the JEP Index rather than being left intact with a closed or withdrawn state.

Thursday, September 9, 2021

Surprisingly High Cost of Java Variables with Capitalized Names

I've read hundreds of thousands or perhaps even millions of lines of Java code during my career as I've worked with my projects' baselines; read code from open source libraries I use; and read code examples in blogs, articles, and books. I've seen numerous different conventions and styles represented in the wide variety of Java code that I've read. However, in the vast majority of cases, the Java developers have used capitalized identifiers for classes, enums and other types and used camelcase identifiers beginning with a lowercase letter for local and other types of variables (fields used as constants and static fields have sometimes had differening naming conventions). Therefore, I was really surprised recently when I was reading some Java code (not in my current project's baseline thankfully) in which the author of the code had capitalized both the types and the identifiers of the local variables used in that code. What surprised me most is how difficult this small change in approach made reading and mentally parsing that otherwise simple code.

The following is a represenative example of the style of Java code that I was so surprised to run across:

Code Listing for DuplicateIdentifiersDemo.java

package dustin.examples.sharednames;

import java.util.Date;
import java.util.List;
import java.util.concurrent.TimeUnit;

import static java.lang.System.out;

/**
 * Demonstrates ability to name variable exactly the same as type,
 * despite this being a really, really, really bad idea.
 */
public class DuplicateIdentifiersDemo
{
    /** "Time now" at instantiation, measured in milliseconds. */
    private final static long timeNowMs = new Date().getTime();

    /** Five consecutive daily instances of {@link Date}. */
    private final static List<Date> Dates = List.of(
            new Date(timeNowMs - TimeUnit.DAYS.toMillis(1)),
            new Date(timeNowMs),
            new Date(timeNowMs + TimeUnit.DAYS.toMillis(1)),
            new Date(timeNowMs + TimeUnit.DAYS.toMillis(2)),
            new Date(timeNowMs + TimeUnit.DAYS.toMillis(3)));

    public static void main(final String[] arguments)
    {
        String String;
        final Date DateNow = new Date(timeNowMs);
        for (final Date Date : Dates)
        {
            if (Date.before(DateNow))
            {
                String = "past";
            }
            else if (Date.after(DateNow))
            {
                String = "future";
            }
            else
            {
                String = "present";
            }
            out.println("Date " + Date + " is the " + String + ".");
        }
    }
}

The code I encountered was only slightly more complicated than that shown above, but it was more painful for me to mentally parse than it should have been because of the naming of the local variables with the exact same names as their respective types. I realized that my years of reading and mentally parsing Java code have led me to intuitively initially think of identifiers beginning with a lowercase letter as variable names and identifiers beginning with an uppercase letter as being type identifiers. Although this type of instinctive assumption generally allows me to more quickly read code and figure out what it does, the assumption in this case was hindering me as I had to put special effort into not allowing myself to think of some occurrences of "String" and "Date" as variables names and occurrences as class names.

Although the code shown above is relatively simple code, the unusual naming convention for the variable names makes it more difficult than it should be, especially for experienced Java developers who have learned to quickly size up code by taking advantage of well-known and generally accepted coding conventions.

The Java Tutorials section on "Java Language Keywords" provides the "list of keywords in the Java programming language" and points out that "you cannot use any of [the listed keywords] as identifiers in your programs." It also mentions that literals (but not keywords) true, false, and null also cannot be used as identifiers. Note that this list of keywords includes the primitive types such as boolean and int, but does not include identifiers of reference types such as String, Boolean, and Integer.

Because very close to all Java code that I had read previously used lowercase first letters for non-constant, non-static variable names, I wondered if that convention is mentioned in the Java Tutorial section on naming variables. It is. That "Variables" section states: "Every programming language has its own set of rules and conventions for the kinds of names that you're allowed to use, and the Java programming language is no different. ... If the name you choose consists of only one word, spell that word in all lowercase letters. If it consists of more than one word, capitalize the first letter of each subsequent word. The names gearRatio and currentGear are prime examples of this convention."

Conclusion

I've long been a believer in conventions that allow for more efficient reading and mental parsing of code. Running into this code with capitalized first letters for its camelcase variable name identifiers reminded me of this and has led me to believe that the greater the general acceptance of a convention for a particular language, the more damaging it is to readability to veer from that convention.

Saturday, February 27, 2021

Java NullPointerException Avoidance and Enhancement Tactics

An encountered NullPointerException can be a useful mechanism for highlighting when a certain code flow or certain data has led to unexpected results (and the messages provided by NullPointerException are much improved with JDK 15). However, there are other times when the presence of null is not an exceptional condition and for those such cases there are several tactics that can be used to easily and cleanly avoid an unwanted NullPointerException. Even when the occurrence of a NullPointerException helps identify problems, there are other tactics we can use to make the most of these opportunities.

The code samples featured in this post are part of class NullSafeTactics and its full source code is available on GitHub.

Contents

Elegantly Avoiding Unnecessary NullPointerExceptions

Implicit Java String Conversion

There are often times when we want the string representation of something that is potentially null and we do not want the access of that string representation to result in a NullPointerException. An example of this is when we log certain conditions and the context we include in the logged message includes a variable or field that is null. It is highly unlikely in such a case that we want a NullPointerException to be possibly thrown during the attempted logging of some potentially different condition. Fortunately, Java's string conversion is often available in these situations.

Even when the field variable NULL_OBJECT of type Object is null, the following code will NOT result in a NullPointerException thanks to Java's string conversion handling a null implicitly and converting it to the "null" string instead.

/**
 * Demonstrates that Java string conversion avoids {@link NullPointerException}.
 */
public void demonstrateNullSafeStringConversion()
{
   executeOperation(
      "Implicit Java String Conversion",
      () -> "The value of the 'null' object is '" + NULL_OBJECT + "'.");
}

The output from running the above code snippet demonstrates that the NullPointerException does not get thrown.

Feb 25, 2021 9:26:19 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
INFO: Demonstration 'Implicit Java String Conversion' completed without exception!

The implicit Java string conversion avoided a NullPointerException. When toString() is explicitly called on that same null, a NullPointerException is encountered. This is shown in the next code listing and the output it leads to is shown after the code listing.

/**
 * Demonstrates that explicit {@link Object#toString()} on {@code null} leads to
 * {@link NullPointerException}.
 */
public void demonstrateNullUnsafeExplicitToString()
{
   executeOperation(
      "Unsafe Explicit toString() Invocation on null",
      () -> "The value of the 'null' object is '" + NULL_OBJECT.toString() + "'.");
}
Feb 25, 2021 9:32:06 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
SEVERE: Exception encountered while trying to run operation for demonstration 'Unsafe Explicit toString() Invocation on null': java.lang.NullPointerException: Cannot invoke "Object.toString()" because "dustin.examples.nullsafe.tactics.NullSafeTactics.NULL_OBJECT" is null

Note that these examples in this post have been executed with a JDK 17 early access release, so the NullPointerExceptions shown in this post benefit from the better NPE messages introduced with JDK 14 (and are enabled by default since JDK 15).

Null-safe String Representation with String.valueOf(Object)

Allowing Java's implicit string conversion to represent null as the "null" string is the cleanest and easiest way to handle null when constructing strings. However, there are many times when we need a string representation of a Java object when implicit string conversion is not available. In such cases, String.valueOf(Object) can be used to achieve functionality similar to the implicit string conversion. When an object is passed to String.valueOf(Object), that method will return the results of the object's toString() if that object is not null or will return the "null" string if the object is null.

The following code listing demonstrates String.valueOf(Object) in action and the output from running that code is shown after the code listing.

/**
 * Demonstrates that {@link String#valueOf(Object)} will render {@code null} safely
 * as "null" string.
 *
 * In many cases, use of {@link String#valueOf(Object)} is unnecessary because Java's
 * string conversion will perform the same effect. {@link String#valueOf(Object)} is
 * necessary when Java is not able to implicitly convert to a {@link String}.
 *
 * See also https://marxsoftware.blogspot.com/2009/04/value-of-stringvalueof.html.
 */
public void demonstrateNullSafeStringValueOf()
{
   executeOperation(
      "Null-safe String Representation with String.valueOf(Object)",
      () -> "The value of the 'null' object is '" + String.valueOf(NULL_OBJECT) + "'.");
}
Feb 25, 2021 10:05:52 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
INFO: Demonstration 'Null-safe String Representation with String.valueOf(Object)' completed without exception!

There are several overloaded versions of String#valueOf accepting parameter types other than Object, but they all behave similarly.

Null-safe String Representation with Objects.toString(Object)

The Objects class provides several methods to allow for elegant handling of potential nulls. One of these, Objects.toString(Object) works exactly like the just-discussed String.valueOf(Object). In fact, as described in the post "String.valueOf(Object) versus Objects.toString(Object)", the Objects.toString(Object) method delegates to the String.valueOf(Object) method.

The following code listing demonstrates use of Objects.toString(Object) and the output from running it follows the code listing.

/**
 * Demonstrates that {@link Objects#toString(Object)} will render {@code null} safely
 * as "null" string.
 *
 * In many cases, use of {@link Objects#toString(Object)} is unnecessary because Java's
 * string conversion will perform the same effect. {@link Objects#toString(Object)} is
 * necessary when Java is not able to implicitly convert to a {@link String}.
 */
public void demonstrateObjectsToString()
{
   executeOperation(
      "Null-safe String Representation with Objects.toString(Object)",
      () -> "The value of the 'null' object is '" + Objects.toString(NULL_OBJECT) + "'.");
}
Feb 25, 2021 10:19:52 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
INFO: Demonstration 'Null-safe String Representation with Objects.toString(Object)' completed without exception!

I tend to use String.valueOf(Object) instead of Objects.toString(Object) because the latter calls the former anyway and because there are overloaded versions of String#valueOf.

Null-safe String Representation with Objects.toString(Object, String)

The approaches covered so far in this post (implicit string conversion, String#valueOf methods, and Objects.toString(Object)) all result in the "null" string when a null is presented to them. There are times when we may prefer to have something other than the "null" string be presented as the string representation of null. An example of this is when we want to return an empty string from a method rather than returning null from a method. The following code listing demonstrates using Objects.toString(Object, String) to have an empty string be provided when first passed-in argument turns out to be null.

/**
 * Demonstrates that {@link Objects#toString(Object, String)} will render {@code null}
 * potentially safely as the "default" string specified as the second argument.
 *
 * In many cases, use of {@link Objects#toString(Object, String)} is unnecessary because
 * Java's string conversion will perform the same effect. {@link Objects#toString(Object)}
 * is necessary when Java is not able to implicitly convert to a {@link String} or when
 * it is desired that the string representation of the {@code null} be something other
 * than the "null" string.
 */
public void demonstrateObjectsToStringWithDefault()
{
   executeOperation(
      "Null-safe String Representation with Objects.toString(Object,String) Using Empty String Default",
      () -> "The value of the 'null' object is '" + Objects.toString(NULL_OBJECT, "") + "'.");
}
Feb 25, 2021 10:33:16 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
INFO: Demonstration 'Null-safe String Representation with Objects.toString(Object,String) Using Empty String Default' completed without exception!

Default Value Replacement of null for Any Object

The JDK-provided methods covered so far are useful for safely acquiring string representation of objects that might be null. Sometimes, we may want to handle a potential instance that might be null of a class other than String. In that case, the Objects.requireNonNullElse(T, T) method allows specification of a default value that should be used if the object in question (first parameter to the method) is null. This is demonstrated with the following code listing and its accompanying output that follows it.

/**
 * Demonstrates that {@link Objects#requireNonNullElse(Object, Object)} will render
 * {@code null} safely for any potential {@code null} passed to it by returning the
 * supplied default instead when the object in question is {@code null}. Two
 * examples are included in this method's demonstration:
 * 
    *
  1. {@code null} {@link Object} safely rendered as custom supplied default "null" string
  2. *
  3. {@code null} {@link TimeUnit} safely rendered as custom supplied default {@link TimeUnit#SECONDS}
  4. *
* * In many cases, use of {@link Objects#requireNonNullElse(Object, Object)} is not * necessary because Java's string conversion will perform the same effect. * {@link Objects#requireNonNullElse(Object, Object)} is necessary when Java is not * able to implicitly convert to a {@link String} or when the potentially {@code null} * object is not a {@link String} or when the object to have a default returned * when it is {@code null} is of class other than {@link String}. */ public void demonstrateNullSafeObjectsRequireNonNullElse() { executeOperation( "Null-safe String Representation with Objects.requireNonNullElse(Object, Object)", () -> "The value of the 'null' object is '" + Objects.requireNonNullElse(NULL_OBJECT, "null") + "'"); executeOperation("Null-safe TimeUnit access with Objects.requireNonNullElse(Object, Object)", () -> "The value used instead of 'null' TimeUnit is '" + Objects.requireNonNullElse(NULL_TIME_UNIT, TimeUnit.SECONDS) + "'"); }
Feb 28, 2021 2:54:45 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
INFO: Demonstration 'Null-safe String Representation with Objects.requireNonNullElse(Object, Object)' completed without exception!
Feb 28, 2021 2:54:45 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
INFO: Demonstration 'Null-safe TimeUnit access with Objects.requireNonNullElse(Object, Object)' completed without exception!

Another Objects method with slightly different name (requireNonNullElseGet(T, Supplier<? extends T>)) allows the default that will be used in place of null to be specified using a Supplier. The advantage of this approach is that the operation used to compute that default value will only be executed if the object is null and the cost of executing that Supplier is NOT incurred if the specified object is null (Supplier deferred execution).

Comparing enums Safely

Although Java enums can be compared for equality using Enum.equals(Object), I prefer to use the operators == and != for comparing enums because the latter is null-safe (and arguably makes for easier reading).

The code listing and associated output that follow demonstrate that comparing enums with == is null-safe but comparing enums with .equals(Object) is NOT null-safe.

/**
 * Demonstrates that comparing a potentially {@code null} enum is
 * {@code null}-safe when the {@code ==} operator (or {@code !=}
 * operator) is used, but that potentially comparing a {@code null}
 * enum using {@link Enum#equals(Object)} results in a
 * {@link NullPointerException}.
 *
 * See also https://marxsoftware.blogspot.com/2011/07/use-to-compare-java-enums.html.
 */
public void demonstrateEnumComparisons()
{
   executeOperation(
      "Using == with enums is null Safe",
      () -> NULL_TIME_UNIT == TimeUnit.MINUTES);
   executeOperation(
      "Using .equals On null Enum is NOT null Safe",
      () -> NULL_TIME_UNIT.equals(TimeUnit.MINUTES));
}
INFO: Demonstration 'Using == with enums is null Safe' completed without exception!
Feb 28, 2021 4:30:17 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
SEVERE: Exception encountered while trying to run operation for demonstration 'Using .equals On null Enum is NOT null Safe': java.lang.NullPointerException: Cannot invoke "java.util.concurrent.TimeUnit.equals(Object)" because "dustin.examples.nullsafe.tactics.NullSafeTactics.NULL_TIME_UNIT" is null
Feb 28, 2021 4:30:17 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation

Comparing Objects Safely with Known Non-null Object on LHS of .equals(Object)

When we know that at least one of two objects being compared is definitely NOT null, we can safely compare the two objects (even if the other one may be null), by calling Object.equals(Object) against the known non-null object. There still is an element of risk here if the class which you're calling .equals(Object) against has its Object.equals(Object) method implemented in such a way that passing in a null argument leads to a NullPointerException. However, I've never encountered a JDK class or even custom class that has made that mistake (and it is a mistake in my opinion to have a Object.equals(Object) overridden method not be able to handle a supplied null and simply return false in that case instead of throwing NullPointerException). The tactic of calling .equals(Object) against the known non-null object is demonstrated in the next code listing and associated output.

/**
 * Demonstrates that comparisons against known non-{@code null} strings can be
 * {@code null}-safe as long as the known non-{@code null} string is on the left
 * side of the {@link Object#equals(Object)} method ({@link Object#equals(Object)})
 * is called on the known non-{@code null} string rather than on the unknown
 * and potential {@code null}.
 */
public void demonstrateLiteralComparisons()
{
   executeOperation(
      "Using known non-null literal on left side of .equals",
         () -> "Inspired by Actual Events".equals(NULL_STRING));
   executeOperation(
      "Using potential null variable on left side of .equals can result in NullPointerExeption",
      () -> NULL_STRING.equals("Inspired by Actual Events"));
}
Feb 28, 2021 4:46:20 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
INFO: Demonstration 'Using known non-null literal on left side of .equals' completed without exception!
Feb 28, 2021 4:46:20 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
SEVERE: Exception encountered while trying to run operation for demonstration 'Using potential null variable on left side of .equals can result in NullPointerExeption': java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "dustin.examples.nullsafe.tactics.NullSafeTactics.NULL_STRING" is null

Although it was specifically String.equals(Object) demonstrated above, this tactic applies to instances of any class as long as the class's .equals(Object) method can gracefully handle a supplied null (and I cannot recall ever encountering one that didn't handle null).

Case Insensitive Comparison of Strings Safely with Known Non-null String on LHS of .equals(Object)

Placing the known non-null object on the left side of the .equals(Object) call is a general null-safe tactic for any object of any type. For String in particular, there are times when we want a null-safe way to compare two strings without regard to the case of the characters in the strings (case insensitive comparison). The String.equalsIgnoreCase(String) method works well for this and will be a null-safe operation if we use a known non-null String on the left side of that method (method called against the known non-null String).

The code listing and associated output that follow demonstrate null-safe use of String.equalsIgnoreCase(String).

/**
 * Demonstrates that case-insensitive comparisons against known non-{@code null}
 * strings can be {@code null}-safe as long as the known non-{@code null} string
 * is on the left side of the {@link Object#equals(Object)} method
 * ({@link Object#equals(Object)}) is called on the known non-{@code null} String
 * rather than on the unknown potential {@code null}).
 */
public void demonstrateLiteralStringEqualsIgnoreCase()
{
   executeOperation(
      "String.equalsIgnoreCase(String) is null-safe with literal string on left side of method",
      () -> "Inspired by Actual Events".equalsIgnoreCase(NULL_STRING));
   executeOperation(
      "Using potential null variable of left side of .equalsIgnoreCase can result in NPE",
      () -> NULL_STRING.equalsIgnoreCase("Inspired by Actual Events"));
}
Feb 28, 2021 7:03:42 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
INFO: Demonstration 'String.equalsIgnoreCase(String) is null-safe with literal string on left side of method' completed without exception!
Feb 28, 2021 7:03:42 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
SEVERE: Exception encountered while trying to run operation for demonstration 'Using potential null variable of left side of .equalsIgnoreCase can result in NPE': java.lang.NullPointerException: Cannot invoke "String.equalsIgnoreCase(String)" because "dustin.examples.nullsafe.tactics.NullSafeTactics.NULL_STRING" is null

These last two demonstrations used literal strings as the "known non-null" strings against which methods were called, but other strings and objects could also be used. Constants and known previously initialized fields and variables are all candidates for the objects against which the comparison methods can be called safely as long as it's known that those fields and variables can never be changed to null. For fields, this condition is only guaranteed if that field is always initialized to a non-null value with the instance and is immutable. For variables, this condition is only guaranteed if that variable is initialized to an unknown value and is final. There are many "in between" cases where it is most likely that certain objects are not null, but guarantees cannot be made. In those cases, it is less risky to explicitly check each object being compared for null before comparing them with .equals(Object) or to use the Objects.equals(Object, Object) method, which is covered next.

Safely Comparing Objects When Neither is Known to be Non-null

The Objects.equals(Object, Object) method is a highly convenient way to compare two objects for equality when we don't know whether either or both might be null. This convenience method's documentation explains its behavior and it's probably what most of u would do if writing this code ourselves, "Returns true if the arguments are equal to each other and false otherwise. Consequently, if both arguments are null, true is returned. Otherwise, if the first argument is not null, equality is determined by calling the equals method of the first argument with the second argument of this method. Otherwise, false is returned."

This is demonstrated in the next code listing and associated output.

/**
 * Demonstrates that comparisons of even potential {@code null}s is safe
 * when {@link Objects#equals(Object, Object)} is used.
 */
public void demonstrateObjectsEquals()
{
   executeOperation(
      "Using Objects.equals(Object, Object) is null-safe",
      () -> Objects.equals(NULL_OBJECT, LocalDateTime.now()));
}
Feb 28, 2021 5:11:19 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
INFO: Demonstration 'Using Objects.equals(Object, Object) is null-safe' completed without exception!

I like to use the Objects.equals(Object, Object) to quickly build my own class's .equals(Objects) methods in a null-safe manner.

The method Objects.deepEquals(Object, Object) is not demonstrated here, but it's worth pointing out its existence. The method's documentation states, "Returns true if the arguments are deeply equal to each other and false otherwise. Two null values are deeply equal. If both arguments are arrays, the algorithm in Arrays.deepEquals is used to determine equality. Otherwise, equality is determined by using the equals method of the first argument."

Null-safe Hashing

The methods Objects.hashCode(Object) (single object) and Objects.hash(Object...) (sequences of objects) can be used to safely generate hash codes for potentially null references. This is demonstrated in the following code listing and associated output.

/**
 * Demonstrates that {@link Objects#hashCode(Object)} is {@code null}-safe.
 */
public void demonstrateObjectsHashCode()
{
   executeOperation(
      "Using Objects.hashCode(Object) is null-safe",
      () -> Objects.hashCode(NULL_OBJECT));
}

/**
 * Demonstrates that {@link Objects#hash(Object...)} is {@code null}-safe.
 */
public void demonstrateObjectsHash()
{
   executeOperation(
      "Using Objects.hash(Object...) is null-safe",
      () -> Objects.hash(NULL_OBJECT, NULL_STRING, NULL_TIME_UNIT));
}
Feb 28, 2021 5:11:19 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
INFO: Demonstration 'Using Objects.hashCode(Object) is null-safe' completed without exception!
Feb 28, 2021 5:11:19 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
INFO: Demonstration 'Using Objects.hash(Object...) is null-safe' completed without exception!

These methods can be convenient for generating one's own null-safe hashCode() methods on custom classes.

It's also important to note that there is a warning in the documentation that the hash code generated by Objects.hash(Object...) for a single supplied Object is not likely to be the same value as a hash code generated for that same Object when calling the Object's own hashCode() method or when calling Objects.hashCode(Object) on that Object.

Elegantly Handling Useful NullPointerExceptions

The tactics described and demonstrated so far were primarily aimed at avoiding NullPointerException in situations where we fully anticipated a reference being null but that presence of a null is in no way exceptional and so we don't want any exception (including NullPointerException) to be thrown. The remainder of the descriptions and examples in this post will focus instead of situations where we want to handle a truly unexpected (and therefore exceptional) null as elegantly as possible. In many of these cases, we do NOT want to preclude the NullPointerException from being thrown because its occurrence will communicate to us some unexpected condition (often bad data or faulty upstream code logic) that we need to address.

The improved NullPointerException messages have made unexpected NullPointerExceptions far more meaningful. However, we can often take a few additional tactics to further improve the usefulness of the NullPointerException that is thrown when we run into an unanticipated null. These tactics include adding our own custom context details to the exception and throwing the exception early so that a bunch of logic is not performed needlessly that may also need to be reverted.

Controlling When and What Related to Unexpected null

I like to use Objects.requireNonNull(T, String) at the beginning of my public methods that accept arguments which will lead to a NullPointerException if a passed-in argument is null. While a NullPointerException is thrown in either case (either implicitly when an attempt to deference the null or when Objects.requireNonNull(T, String) is called), I like the ability to be able to specify a string with details and context about what's happing when the null is unexpectedly encountered.

The Objects.requireNonNull(T) method does not allow one to specify a string with additional context, but it is still a useful guard method. Both of these methods allow the developer to take control of when a NullPointerException will be thrown for the unexpected null and this control allows the developer to preclude unnecessary logic from being performed. We'd rather not waste time/cycles on something that is going to lead to that exception anyway and we can often choose places in the code where it's cleaner to check for null and throw the exception to avoid having to "undo" pr "revert" logic that has been performed.

The following code listing and associated output demonstrate both of these methods in action.

/**
 * Demonstrates using {@link Objects#requireNonNull(Object)} and
 * {@link Objects#requireNonNull(Object, String)} to take control of
 * when an {@link NullPointerException} is thrown. The method accepting
 * a {@link String} also allows control of the context that is provided
 * in the exception message.
 *
 * It is not demonstrated here, but a similar method is
 * {@link Objects#requireNonNull(Object, Supplier)} that allows a
 * {@link Supplier} to be used to provide the message for when an
 * unexpected {@code null} is encountered.
 */
public void demonstrateObjectsRequiresNonNullMethods()
{
   executeOperation(
      "Using Objects.requireNonNull(T)",
      () -> Objects.requireNonNull(NULL_OBJECT));

   executeOperation(
      "Using Objects.requireNonNull(T, String)",
      () -> Objects.requireNonNull(NULL_OBJECT, "Cannot perform logic on supplied null object."));
}
Feb 28, 2021 5:59:42 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
SEVERE: Exception encountered while trying to run operation for demonstration 'Using Objects.requireNonNull(T)': java.lang.NullPointerException
Feb 28, 2021 5:59:42 PM dustin.examples.nullsafe.tactics.NullSafeTactics executeOperation
SEVERE: Exception encountered while trying to run operation for demonstration 'Using Objects.requireNonNull(T, String)': java.lang.NullPointerException: Cannot perform logic on supplied null object.

The output shows us that the method that accepted a String was able to provide that additional context in its message, which can be very useful when figuring out why the unexpected null occurred.

I don't demonstrate it here, but it's worth noting that another overloaded version of this method (Objects.requireNonNull(T, Supplier<String>)) allows a developer to use a Supplier to supply a custom NullPointerException for complete control over the exception that is thrown. The use of the Supplier and its deferred execution means that this exception generation will only be performed when the object is null. One might choose to implement this Supplier as a relatively expensive operation checking various data sources and/or instance values and would not need to worry about incurring that cost unless the unexpected null was encountered.

Other Null-Handling Tactics

There are other tactics that can be used to either avoid unnecessary NullPointerExceptions or to make NullPointerExceptions due to unexpected null more useful. These include explicit checking for null in conditionals and use of Optional.

Conclusion

This post has discussed and demonstrated tactics for using standard JDK APIs to appropriately avoid unnecessary NullPointerExceptions and to more effectively use NullPointerExceptions to indicate unexpected nulls. There are several simple tactics to ensure that expected nulls do not lead to NullPointerException. There are also tactics available to control when a NullPointerException is thrown and what details are provided in it when an unexpected null is encountered.

Thursday, December 31, 2020

Java Predictions for 2021 on Foojay

Geertjan Wielenga has posted "Java Predictions for 2021" on Foojay Today. It is a collection of predictions about Java in 2021 from eight members of the Java community (Almas Baimagambetov, Stephen Chin, Brice Dutheil, Marcus Hirt, Reza Rahman, Matt Raible, Simon Ritter, and me). The predictions are concisely written and it's interesting to see the overlap between them while at the same time seeing how different parts of "Java" are important to different people. In this post, I elaborate a bit more on my predictions that were included in "Java Predictions for 2021".

I provided two somewhat related predictions for Java in 2021:

  • "Records will likely be finalized in 2021 and will be widely popular with Java developers who are fortunate enough to work on a version of the JDK with final (not preview) Record support.
  • The release of the OpenJDK 17 implementation in 2021 (which will be the foundation of Oracle's LTS version and other community members' LTS versions) will motivate many who are already working on versions of the JDK later than JDK 8 to start moving or investigate moving to JDK 17. However, JDK 8 will remain widely popular (probably will still be used by over half of Java developers), creating (in the long term) a bimodal distribution of most commonly used JDK versions (8 and 17)."

Java Records Final in 2021

The prediction that Java Records will be final in 2021 is not a risky one. Records have a been a "preview" feature in JDK 14 (JEP 359) and JDK 15 (JEP 384) and now JEP 395 "proposes to finalize the feature in JDK 16" (which is currently in Rampdown Phase One and is scheduled to be released for General Availability in March 2021). Because Records have been through two preview releases already, it seems unlikely that they won't be final as of JDK 16. In the event that they do need one more release, JDK 17 should be released in October 2021.

And Then There Were Two: JDK 8 and JDK 17

2021 will see the beginning of a move to a bimodial distribution of JDK releases most commnoly used. With JDK 17's likely release in October 2021, we'll likely see many Java shops that have already migrated to a version of JDK later than JDK 8 move to that newly released JDK 17. There have been some nice additions and improvements to OpenJDK (which is the foundation of several different JDK implementations) that have been added in recent versions of the JDK and JDK 17 will be an "LTS" (Long-term Support) release for many of the JDK implementations. As an "LTS" release, JDK 17 will appeal to Java shops that want to only be on versions with long-term support and JDK 17 will be the first since JDK 11 to have this status for many of the JDK implementations.

JDK 8 appears to still be the most widely used release of the Java even in 2020. There are several metrics and andecdotal evidence that suggest this. One example is the JetBrains 2020 Development Ecosystem survey that suggests that 75% of Java developers responding to the survey use JDK 8 (some of these developers use other versions of JDK as well) and the same chart shows 32% of responding Java developers use JDK 11. For reference, the 2019 and 2018 versions of this same survey indicated that 83% and 84% of Java developers used JDK 8 in 2019 and 2018 respectively.

JDK 8 is a version with long-term support (Oracle, for example, offers "extended support" for JDK 8 through December 2030) in several JDK implementations and some shops appear hesitant to move to JDK 9 with its introduced modularity support (and need for libraries and frameworks to support that as well). For those shops that have already migrated to a version of JDK later than JDK 8, it should be relatively easier to migrate to JDK 17. I think that some JDK 8 shops will be motivated to make the "big move" and, while doing that, will jump directly to JDK 17. However, I expect that we'll still see at least half of JDK developers still using JDK 8 even at the end of 2021. For the half of JDK users already using a version later than JDK 8 (not counting users of version of JDK before JDK 8), I think we'll begin to see them migrate to JDK 17 in 2021 and the following year or two. Within the next year or two, I expect most JDK developers will be working with JDK 8 or JDK 17.

There will, of course, be some small pockets of JDK developers using other versions before JDK 8, between JDK 8 and JDK 17 (perhaps because they use a feature or garbage collector no longer available in JDK 17), and newer versions of JDK as they are released in 2022.

"LTS" Among JDK Providers

The following are some roadmaps of various JDK vendors' JDK implementations that provide insight into each vendor's LTS concept. Although "LTS" often is referring to Oracle's plan regarding their JDK implementation built on top of OpenJDK, other JDK vendors have generally treated these "LTS" releases in similar manner.

  • AdoptOpenJDK Support and Release Roadmap
    • Shows "Java 17" as LTS.
    • States, "In addition, every three years one feature release will be designated as a Long Term Supported (LTS) release. We will produce LTS releases for at least four years."
    • States, "As a general philosophy, AdoptOpenJDK will continue to build binaries for LTS releases as long as the corresponding upstream source is actively maintained."
  • Oracle Java SE Support Roadmap
    • States, "For product releases after Java SE 8, Oracle will designate a release, every three years, as a Long-Term-Support (LTS) release. Java SE 11 is an LTS release."
  • Azul Java Support Roadmap
    • References Long Term Support (LTS) and Medium Term Support (MTS) and states, "Releases designated as LTS are those same LTS releases as designated by Oracle and the OpenJDK community."
  • Amazon Corretto
    • "Amazon Corretto 8 & 11 support extended" states, "Amazon is extending long-term support (LTS) for Amazon Corretto 8 from June 2023 to May 2026 and for Amazon Corretto 11 from August 2024 to September 2027. Long-term support (LTS) for Corretto includes security updates and specific performance enhancements released at least quarterly."

Looking Forward to 2021

Most of us are hoping for a better year in 2021 than we've experienced in 2020. The finalization of Java Records and General Availability of JDK 17 in 2021 are going to be significant positive events for Java developers and I'm hoping that these will only be a small representative sample of positive events and advancements that benefit a much wider population in 2021.

Friday, September 25, 2020

foojay - A Place for Friends of OpenJDK

I welcome new sources of information on all things Java and was excited to see what the foojay site has to offer after Geertjan Wielenga (@GeertjanW) pointed it out to me. In this post I highlight some of the most promising characteristics of this site for this seeking new information on Java-related topics.

The foojay.io About page describes the site's name: "A place for friends of OpenJDK." This relatively new Azul-sponsored community site was highlighted in Simon Ritter's 25 April 2020 post "foojay: A Place for Friends of OpenJDK." In that post, Ritter states:

For Java users and developers who depend on OpenJDK builds, finding the relevant highlights of a given update and how they may improve or otherwise impact deployments is no small task. Separating issues that have significance to a user from others would involve going through hundreds of individual issues. Following each into the Java Bug System to review details, and deducing relevance for one’s environment can be quite a challenge. In practice, few Java users or organizations will go through this level of effort.

This is where foojay's user-focused Java and OpenJDK update descriptions come in.

New Bug Fixes and Features in the OpenJDK

As the Ritter quote above demonstrates, one of the primary objectives of the foojay site is to help Java developers understand the various JDK offerings available based on OpenJDK and the changes, fixes, and new features associated with different versions of OpenJDK.

The following describes how to quickly see new features associated with various versions of OpenJDK on the foojay site:

  • The "What's New in OpenJDK?" is front and center on the Foojay page.
    • Provides overview of new features coming to recent versions of OpenJDK.
    • Click on month/year to see versions of OpenJDK with changes in that month.
    • Click on specific version of OpenJDK for that month/year to see change categories such as "Highlights", "All Issues", "Component View", and "Security / CVE View"
    • Click on category tab to see table representation of changes to that OpenJDK version in that category. This table includes different details depending on the category. The "Foojar Commentary" column of the "Highlights" table provides summary details about a particular bug or feature.

Java Version Almanac

The Java Version Almanac section of the Foojay site presents details on various versions from 1.0 to through JDK 16 as of this writing. There are, of course, generally more details in the later versions than the earlier versions, but it's interesting to see how Java and the JDK have changed. The information presented in Foojay's "Java Version Almanac" section is based upon javaalmanac.io, which "is provided by Java Champions Marc R. Hoffmann and Cay S. Horstmann".

OpenJDK Command Line Arguments

Another interesting section of the Foojay site is the section on OpenJDK's supported command-line arguments. This section shows command-line arguments for each version of OpenJDK from Java 6 through Java 16 as of this writing. This section is derived from Chris Newland's chriswhocodes.com site.

Foojay Blog

Since its inception, the foojay site seems to have broaded its scope to cover additional topics related to use of Java and OpenJDK. There are several well-known Java developers/speakers who write posts for the Foojay blog. These include the following authors (not the entire list):

Although many of the bloggers on the Foojay site have their own blog sites with the same blog posts, there is an advantage to being able to see the headlines and blogs of these different authors in one place without needing to go to each author's individual blog.

Things I Like About the Foojay Site

  • Coverage of different versions of Java and OpenJDK
  • Coverage of different implementations of OpenJDK
  • Community-based with contributions from multiple individuals and sites
  • Blog posts focused on Java-related technologies
    • Even though many of these posts are available on the authors' sites, it's easier to see the headlines in a single location and click on stories of interest.
    • I like the strong focus on Java in these blog posts because Java is still my main programming language.
  • Single location for updates on what's new in the Java and OpenJDK world
    • This Azul-sponsored site nicely complements the Oracle-provided Inside Java site and a Java developer is likely to be aware of most major developments in the Javasphere if regularly reviewing these two sites.
  • Significantly fewer advertisements than many software development aggregation sites
    • Although there are mentions of Azul offerings, they are limited and not the invasive pop-up style advertisements plaguing many software development aggregation sites.
  • Backed by Azul
    • Too many blogs and other sources that I like for new information on Java tend to die out quickly and I have greater confidence in the Azul-sponsored Foojay site being around for a while.
    • The Foojay site's association with Azul means that it is associated with an OpenJDK contributor who provides their own implementation based on OpenJDK and who is a participant in the Java Community Process.

Conclusion

We continue to have exciting new things coming soon to a JDK near us. With the 6-month cadence of new OpenJDK releases, it can be difficult to keep up with so much change. The Foojay site is a great tool for Java developers who want to see the most important details regarding what's coming without needing to delve into bug reports and OpenJDK mailing lists.

Thursday, July 2, 2020

jcmd Increasingly the Single JDK Command-line Tool to Rule Them All

What would you miss from the JDK command line tools jinfo, jmap, and jstack if those tools were no longer available?

I've been a fan of the command-line tool jcmd for a long time. I like that it combines features and functions of many other JDK command-line tools in a single tool. I also like that I like its interactive nature that supports my asking the tool what it's capable of for a given JVM process, meaning I don't have to remember exact options when applying its different functions.

Since its inception, jcmd has been taking on functionality from various other command-line JDK tools. In a recent post on the OpenJDK jdk-dev mailing list, Stephen Fitch announced a survey intended to "gather more information and help us evaluate and understand how others are using [three 'j* tools' jinfo, jmap, jstack] in the JDK." The surveyors are "specifically interested in your use-cases and how these tools are effective for you in resolving JVM issues."

Fitch explains the need for this survey:

We are considering deprecation and (eventual) removal of the jinfo, jmap, jstack - (aka “j* tools”) and building out a future foundation for some aspect of serviceability on jcmd, however we don’t have a lot of data about how how these tools are used in practice, especially outside of Oracle.

I'm encouraged that it sounds likely that jcmd will continue to add functionality under its single approach that was formerly available with different tools.

The survey will be open through 15 July 2020 and is an opportunity for Java developers to share their experiences and use cases with the JDK command line tools jinfo, jmap, and jstack so that it's more likely those use cases will be supported in jcmd (if not already supported) in the future. Fitch's post presents the URL for the survey: https://www.questionpro.com/t/AQk5jZhiww

Select jcmd Posts

Here are some of my previous posts regarding jcmd (I have used jcmd in numerous other posts to demonstrate various output as well):

Thursday, June 4, 2020

Tycoon: Ransomware Targeting Java's JIMAGE on Multiple Platforms

The Blackberry Research and Intelligence Team and KPMG's UK Cyber Response Services Team have reported "Threat Spotlight: Tycoon Ransomware Targets Education and Software Sectors." This report outlines the "multi-platform Java ransomware targeting Windows and Linux that has been observed in-the-wild since at least December 2019" and which they've called "Tycoon."

The report provides a high level description of how the Tycoon ransomware is executed: "Tycoon ransomware comes in form of a ZIP archive containing a Trojanized Java Runtime Environment (JRE) build. The malware is compiled into a Java image file (JIMAGE) located at lib\modules within the build directory." The report describes the "sparsely documented" JIMAGE "file format file format that stores custom JRE images which is designed to be used by the Java Virtual Machine (JVM) at runtime." Additional high-level overviews of the JIMAGE file format can be found in JIMAGE - Java Image File Format, How Modules Are Packaged in Java 9, What Is a Custom Runtime Image in Java 9?, So what is a .jimage?, Alan Bateman's description, and slide 49 of JDK 9 Java Platform Module System. The JIMAGE format was introduced with JDK 9 modularity (Project Jigsaw).

Alan Bateman (owner of JEP 220 ["Modular Run-Time Images"])has explained why it's difficult to find documentation on the JIMAGE format: "The format is deliberately not documented and best to assume it will change and evolve over time as it inhales new optimizations." The jdk.jlink module documentation provides very brief mention of the jimage command-line tool and mentions that it is used "for inspecting the JDK implementation-specific container file for classes and resources." It also points out that there is no API for accessing jimage, which is different than for the module's two other tools (jlink and jmod). Similarly, there is no jimage tool reference on the "Java Development Kit Version 14 Tool Specifications" page even though jlink, jmod, and many other tools are featured there.

The previously mentioned report states that "the ransomware is triggered by executing a shell script that runs the Main function of the malicious Java module using the java -m command." The report also talks about Tycoon using Image File Execution Options Injection with Windows registry and then appearing to target Windows and Linux: "The malicious JRE build contains both Windows and Linux versions of this script." This reports adds that Tycoon's "Trojanized Java Runtime Environment (JRE) build" exists as a ZIP file placed in lib/module.

The "Conclusions" section of "Tycoon Ransomware Targets Education and Software Sectors" makes some interesting conclusions based on research observations. In particular, I find it interesting to read why they believe Tycoon in a targeted attack. The authors also point out, "This is the first sample we've encountered that specifically abuses the Java JIMAGE format to create a custom malicious JRE build."

I recommend reading the report "Threat Spotlight: Tycoon Ransomware Targets Education and Software Sectors" directly, but for those interested in summaries and others' observations related to this report, the following may be of interest.

The discovery of Tycoon is likely to bring significantly more attention to JIMAGE than ever before.

Saturday, May 30, 2020

Octopus Scanner: Java Build Tools and Malware

Alvaro Muñoz recently posted "The Octopus Scanner Malware: Attacking the open source supply chain" on the GitHub Security Lab site. I found this post to be interesting for a number of reasons, including its detailed coverage of how the Octopus Scanner malware works and how it was discovered, how the GitHub Security Incident Report Team (SIRT) went about addressing it, how it affected a popular Java IDE, and how GitHub works to detect and address risks to open source software deployed on its site.

Muñoz calls Octopus Scanner "an OSS supply chain malware" and writes that 26 GitHub-hosted open source projects "were backdoored by this malware." In the post, Muñoz describes in detail how Octopus Scanner works. The post in its entirety is worth reading, but here are some highlights describing Octopus Scanner:

  • "Octopus Scanner malware is only interested in the pre-jar and post-jar tasks" of "a NetBeans project build."
  • "The malware disguises itself as an ocs.txt file, but we can easily determine is is actually a Java Archive (JAR) file."
  • "The malware also infected any JAR files that were available in the project, such as dependencies - not necessarily just build artifacts."
  • "The octopus.dat payload is the binary that actually performs the NetBeans build infections."
  • "cache.dat is responsible for backdooring the built classes so that when these classes get executed, they will infect the underlying system."
  • Octopus Scanner is designed to target "UNIX-like systems", MacOS, and Windows.

I found the highly detailed approach to diagnosing Octopus Scanner's behavior in the GitHub post to be fascinating and insightful reading. It was especially insightful to see the tools and methods used to understand better how Octopus Scanner behaves. For example, they used ClassFileTransformer and "a Bytecode manipulation library such as Javassist or ByteBuddy to inject our analysis code" into the "class responsible for decrypting the blob ... right before it actually gets loaded into the JVM."

Besides the interesting details of how Octopus Scanner works and how it was discovered and researched, other interesting insights in this GitHub post are related to the risks open source builds face. Muñoz writes, "Infecting build artifacts is a means to infect more hosts since the infected project will most likely get built by other systems and the build artifacts will probably be loaded and executed on other systems as well." Muñoz adds, "In an OSS context, it gives the malware an effective means of transmission since the affected projects will presumably get cloned, forked, and used on potentially many different systems. The actual artifacts of these builds may spread even further in a way that is disconnected from the original build process and harder to track down after the fact."

Muñoz opens the post and concludes the post with some discussions about this and other attempts at sabotaging open source products and their builds. One chilling thought is included in the Conclusion: "Since the primary-infected users are developers, the access that is gained is of high interest to attackers since developers generally have access to additional projects, production environments, database passwords, and other critical assets. There is a huge potential for escalation of access, which is a core attacker objective in most cases."

Thursday, August 15, 2019

Draft JEP to Remove Deprecated CMS Garbage Collector

In the 10 April 2017 post "Java Garbage Collectors: When Will G1GC Force CMS Out?," I discussed JEP 291 ("Deprecate the Concurrent Mark Sweep (CMS) Garbage Collector") and provided a summary of some of the feedback for and against deprecating (and ultimately removing) the Concurrent Mark Sweep (CMS) Garbage Collector (GC). Earlier this month, a draft JEP called "Remove the Concurrent Mark Sweep Garbage Collector" was created.

The "Summary" section of this draft JEP states, "Remove the Concurrent Mark Sweep (CMS) garbage collector from the set of usable garbage collection algorithms." Although the "draft" status of this JEP means it's not targeted to any specific release, the "Non-Goals" section reassures us that the draft JEP is not intended to remove CMS from any releases prior to the release in which CMS is removed. The post "RFC: JEP: Remove the Concurrent Mark Sweep Garbage Collector" states that the current plan is to target CMS removal for JDK 14.

The draft JEP states in the "Risks and Assumptions" section that this draft to remove CMS GC "might be withdrawn" if another "interested credible contributor in the community will step up for maintenance." However, in the 2+ years since JEP 291 deprecated CMS, no such "interested credible contributor" has offered to maintain CMS. Thomas Schatzl's post related to this JEP describes this better:

There has also always been the option to organize maintenance of CMS in the community, but nobody even stepped up starting to fix the long-standing existing known minor issues CMS (to get contributors to know CMS code and to give us confidence that these persons can take over maintenance of such a large component).

The draft JEP for removing CMS specifically mentions three recommend alternative garbage collectors: the now-default garbage-first (G1), Oracle-provided ZGC, and Red Hat-provided Shenandoah.

There were multiple objections to deprecating CMS when JEP 291 was under review and there are objections now to the idea of removing CMS altogether. Kirk Pepperdine has written that he and others have observed that "CMS overheads are no where near the level of those seen with G1" and they are "now recommending that customers consider Parallel GC as it offers a far better experience than G1."

It looks likely that CMS will be removed as a garbage collection option in a forthcoming JDK release (perhaps even as early as JDK 14). As Kirk Pepperdine expressed in another post on the subject, "At the end of the day, if we want CMS we're going to have to step up and do something about it." So far, it doesn't look like anyone's wanted CMS badly enough to do something about it (which might include working with their preferred JDK provider to have it supported in that provider's JDK).