Remember that arrays are objects and you what you declare as final is the reference to the array. Declaring an array reference final simply means that that array reference cannot be used to refer to a different array. It does not mean that the elements in the array are final.
Arrays are intrinsically mutable. You cannot detach the notion of mutability from an array, however, you can abandon arrays with minimal effort. The simplest way to do that is with an interface declaration:
...so long as you omit the "set" operation from the contract (which exists only to have an observable side-effect on the "get" operation - which is impure), you now have an immutable array. You could of course, write E such that it is mutable, however, I have ignored that for now.
The problem you are observing - and you are certainly not the first by a long shot - is what I have called "violation of perfect symbiosis of contractual operation", which is a side-effect that is forever tied to the notion of static contract inheritance (and OO as we know it). Simply, but not accurately, Java declares three operations that are symbiotic (forever in existence with each other), however, you have come up with a legitimate use case where you wish to remove one of those operations (the "set"). This is what "design patterns" are often used for - for example, "wrapping" types with less contract. Ultimately, they are workarounds to a language defect. In fact, if you manage to perform the required mindshift (the hard bit), you can observe that some extraordinary amount (99.9%?) of software development with Java is working around its shortcomings, which begs the question, "what are the consequences if these shortcomings were to be eliminated?".
ContractualJ mandates only one operation per interface (still not optimal ala static contract inheritance) and introduces the Sequence interface which attempts a minimal impact on clients for coming up with legitimate use cases that force a work around to static contract inheritance. That is, you can abandon arrays with a dependency on ContractualJ - a trade off that I often opt for in production.