Correctly detect Big Endian on Solaris

The current code fails to detect Big Endian SPARC on Solaris. The
reason for this is that __BYTE_ORDER is defined on Solaris, however
because we're trying to avoid a case on Linux where __BYTE_ORDER and
__BIG_ENDIAN will always be defined, we're only defining
SHA1DC_BIGENDIAN if the two are equivalent.

That's not the case on Solaris, and tweaking this code easily lands
you in the case where you ruin either Linux or Solaris, because on
Linux those two are always defined.

There's probably some really tricky middle ground here that would
auto-detect this without checking the OS, e.g. I know _BIG_ENDIAN is
defined on Solaris, but probably not on Linux (but all versions? who
knows?). Liam R. Howlett <Liam.Howlett@oracle.com> reports that this
works, and it works for me too on Linux x86_64:

    diff --git a/sha1dc/sha1.c b/sha1dc/sha1.c
    index facea1bb5..8c95c64ce 100644
    --- a/sha1dc/sha1.c
    +++ b/sha1dc/sha1.c
    @@ -38,7 +38,8 @@

     #if (defined(_BYTE_ORDER) || defined(__BYTE_ORDER) || defined(__BYTE_ORDER__))

    -#if ((defined(_BYTE_ORDER) && (_BYTE_ORDER == _BIG_ENDIAN)) || \
    +#if ((defined(__BYTE_ORDER__) && defined(_BIG_ENDIAN)) || \
    +     (defined(_BYTE_ORDER) && defined(_BIG_ENDIAN) ) || \
          (defined(__BYTE_ORDER) && (__BYTE_ORDER == __BIG_ENDIAN)) || \
          (defined(__BYTE_ORDER__) && (__BYTE_ORDER__ == __BIG_ENDIAN__)) )
     #define SHA1DC_BIGENDIAN

As an aside, this whole thing looks like it may have worked on Solaris
before a24eef5 ("rewrote Endianness selection", 2017-05-29), but of
course other stuff was broken before then.

Let's just take the easy way out here and manually detect Solaris,
they're unlikely to add another Big Endian processor any time soon,
and then the rest of this logic can stop worrying about being paranoid
about Solaris.

1 files changed, 24 insertions, 0 deletions

diff --git a/lib/sha1.c b/lib/sha1.c
index facea1b..3f01599 100644
--- a/lib/sha1.c
+++ b/lib/sha1.c
@@ -36,6 +36,28 @@
 #undef SHA1DC_BIGENDIAN
 #endif
 
+#ifdef sun
+
+#ifdef __sparc
+/*
+ * Why not do this generically? Because Linux at least will define
+ * __BIG_ENDIAN but we're only Big Endian if __BYTE_ORDER is
+ * equivalent to __BIG_ENDIAN, but on Solaris SPARC _BIG_ENDIAN is
+ * defined without any value. Thus just checking if _BIG_ENDIAN is
+ * defined on Solaris SPARC works, but we can't just check if it's
+ * defined because on Linux x86 it'll be defined, just as
+ * __BIG_ENDIAN.
+ *
+ * There's probably some easy way out of this, but let's just take the
+ * easy way out here and treat Solaris specially. This is the Oracle
+ * documented way to check for Solaris SPARC. See
+ * http://www.oracle.com/technetwork/server-storage/solaris/portingtosolaris-138514.html
+ */
+#define SHA1DC_BIGENDIAN
+#endif
+
+#else
+
 #if (defined(_BYTE_ORDER) || defined(__BYTE_ORDER) || defined(__BYTE_ORDER__))
 
 #if ((defined(_BYTE_ORDER) && (_BYTE_ORDER == _BIG_ENDIAN)) || \
@@ -55,6 +77,8 @@
 
 #endif
 
+#endif
+
 #if (defined(SHA1DC_FORCE_LITTLEENDIAN) && defined(SHA1DC_BIGENDIAN))
 #undef SHA1DC_BIGENDIAN
 #endif