Mike London

Bartender
+ Follow
since Jul 12, 2002
Cows and Likes
Cows
Total received
17
In last 30 days
0
Total given
0
Likes
Total received
52
Received in last 30 days
0
Total given
20
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Rancher Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Mike London

Carey Brown wrote:And how did the new file look any different?



Hey Carey,

Nope, both were ostensibly just regular text files. I have no idea.

Appreciate your reply.

-- mike
1 month ago
I think the problem was the Excel export to CSV.

I took all the values from that CSV and copied them into a new file and it worked fine.

Strange, right?

Thanks to all!

- mike
1 month ago
I'm reading from a CSV file with lines like this:

2,2021-10-11,12,29.00,

When I read the first value using the Scanner class, the debugger says it's "2", but it's a byte array. Thus, when I try to Integer.parseInt(), it fails with a number format exception.

I also tried code like this:



but that also fails.

I've really not used the Scanner class that much so I would appreciate any suggestions.

Thanks,
- mike
1 month ago
I found the solution.

Starting the service with this command line works
$ java -jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 service.jar

Although the following command works with other REST functions when loading the JAR, it didn't work with DB upload: java -jar service.jar

Hope this helps somebody.

- mike

2 months ago
For some odd reason, I can run a simple REST service in my IDE and the upload to DropBox works.

However, in the built JAR file, it crashes badly (doesn't return Exception info, just "server error.") on the "uploadAndFinish()" line of code below.

Any ideas why this might not work in the JAR but would work in the IDE?

Below, when running from the JAR file, the "before   FileMetadata metadata" is the last message returned.

I'm using maven, version 4.0.1 of the DropBox Core library, but the same problem happened with the previous versoin.

Thanks in advance,

-- mike

2 months ago

Tim Holloway wrote:One other thing and I should have said it earlier. Rather than using LOCAL_SYSTEM (which I think is actually ACLs), I'd recommend creating a distinct user account for Tomcat. You could even name it something like "tomcat".

I'm pretty sure that this has been a common practice for Windows Services running Tomcat. It is certainly common on Unix/Linux systems.

It's more accessible than a built-in identity and you can fine-tune its access rights, giving you a more secure system overall.



That's a good tip.

Thanks again, Tim!  

-- mike
2 months ago

Rob Spoor wrote:The LOCAL SYSTEM account has a different way of interacting with environment variables. It probably doesn't have any, or at least not as many as regular logged in users have. Redefining the necessary environment variables probably would have solved your problem. Is that by any chance what you did?

I strongly suggest that you use ProcessBuilder. It has a much richer API for setting up processes. No annoying things like a String[] for the environment, but a Map instead. And of course the redirecting of the error stream, which can be essential to prevent deadlocks. For more information on that, read the classic When Runtime.exec() won't article. By doing the redirect there is only one stream to read from, and you don't need any additional threads. Just read from the single stream until it's exhausted.



Thanks Bob. Appreciate that suggestion and will look to incorporate it soon.  I did incorporate the error stream reader as suggested to log any problems found with the process exec(). That works great also (thanks again!) and helped me fix the remaining problem running Tomcat as a service for this usage.

The way I solved the issue was to pass the authentication tokens with the Python program itself when creating the s3.client. That worked fine. It was just that the sample AWS Transcription python example did not have those two parameters in its example when it created the s3 client and I just ran into the problem of the Environment variables under Tomcat running as a service.

I really want to thank Tim again for his excellent suggestion on how to get Tomcat running as a service to be debuggable on port 8000. I can't tell you how many hours that will save me in the future!

Thanks again for all the beyond-superb help and suggestions!

-- mike
2 months ago

Mike London wrote:

Tim Holloway wrote:Mmmm, yes. Python has an entire ecosystem all its own including virtual environments, PYTHONPATHS and more..

I think actually your best bet would be run create a BAT file that runs the Python interpreter and Runtime.exec() the BAT file. I've often avoided a world of pain from that little bit of extra overhead.



If I log in as "Mike" into Tomcat, it works!

Now I need to figure out where the "LOCAL SYSTEM" account is so I can install boto3 dependencies.

I think you should get "Person of the Year" for all your great help!!!

You're the best.

Thanks again!

-- mike



IT'S ALL WORKING NOW!!!

Thank you Thank you Thank you.

-- mike
2 months ago

Tim Holloway wrote:Mmmm, yes. Python has an entire ecosystem all its own including virtual environments, PYTHONPATHS and more..

I think actually your best bet would be run create a BAT file that runs the Python interpreter and Runtime.exec() the BAT file. I've often avoided a world of pain from that little bit of extra overhead.



If I log in as "Mike" into Tomcat, it works!

Now I need to figure out where the "LOCAL SYSTEM" account is so I can install boto3 dependencies.

I think you should get "Person of the Year" for all your great help!!!

You're the best.

Thanks again!

-- mike
2 months ago

Tim Holloway wrote:Unfortunately, I'm not set up to check it out directly, but you definitely should be able to connect via remote debug.

The Java debugger is built into the JVM itself and is activated via "java" command line options. I think it's under "agent transport", but I'd have to go back and read the docs. Anyway, the debugger in an IDE is simply a GUI client for the JVM debugger. The debugger itself is actually a text command remote shell, so you don't even actually need an IDE, although it's more fun that way.

Knowing this, you can see that yes, remote debugging should be possible and in fact:

https://stackoverflow.com/questions/1406320/starting-tomcat-with-remote-debugging-jdwp-when-installed-as-a-windows-service



OK, much closer now! The tip on how to debug the service was the key!

The first problem I immediately found was that Java (running under Tomcat as a Service) could not find "python". That problem was easy enough to fix by adding a path to point to that python directory where the executable is. Now the python program starts to run, finally!

The current problem is the import "import boto3" (the AWS library) is not resolving.

Using the same methodology, I put the path to the boto3 library into the System Environment variables, like this:

C:\Users\mike\AppData\Roaming\Python\Python39\site-packages

(the AWS boto3 is in this folder)

However, when the Python program runs (under Tomcat with Windows Service), I now get this:

 File "C:\pythonscripts\aws_transcribe.py", line 17, in <module>
   import boto3
ModuleNotFoundError: No module named 'boto3'



If you have any ideas, I'd appreciate it.

Not sure how to get around this one, but I'll keep searching...

Appreciate all your help!

-- mike
2 months ago

Tim Holloway wrote:I just re-read the docs and although there are many ways to invoke Runtime.exec they're all very definite about NPE. The only thing short of a fault in the JVM that can cause that is if the command, command array or environment array (if supplied) have null values.

You may be suffering from "too much knowledge", which is my #1 way to spend days on end on an otherwise trivial problem. The "too much knowledge" effect comes from when what's on the screen (or printed page) and what you're reading are not the same thing because you "know" what the values should be and don't see the difference. Often the best solution for me has been to find a junior programmer who can read without prejudice and to whom, therefore, the fault is obvious.

Which is also why debuggers are good, since they can show you what the actual values are at or before the point of failure as opposed to what you "know" they're supposed to be.You can still mist defective data values in a debugger, but since the context is different, you're more likely to see something that you didn't see in the actual code.



Right.

There's no way I know of to debug a service "remotely" when run from the Windows Service manager.

The way I do it is to start Tomcat (a separate instance) using: "catalina jpda start". Then I can connect to port 8000 and step through from the IDE.

If you know how I can connect to the live instance running under Tomcat via the Service Manager, please let me know.

When I did connect using port 8000 on the standalone instance, the values were OK. Not sure what they might be under Tomcat run from Windows service.

Please let me know.

Thanks!

- mike

2 months ago

Tim Holloway wrote:No, I'm not overlooking the different Tomcat environments. But I don't expect the "working directory" for Tomcat as a service to be the same for example. It's a personal fault of mine that I don't like to "fix" things, I like to find out what's wrong. Becaase I hate it when things I "fixed" break when assumptions the fix makes have changed.

Anyway, you didn't confirm that process is the null object, but if it is, the JavaDocs are clear. Anything other than a bad cmdArray should not return null and if it does, your JVM is defective. Meaning you need to find a properly-implemented one.



I'm trying to figure out what's wrong also, but considering it works fine (as expected) standalone vs. not working if started from a service, then the JVM could be the next thing to swap out.

Thanks Tim. I'll report back.

-- mike
2 months ago

Tim Holloway wrote:OK. Let me localize the point of failure. You have:


My deduction is that "process" is coming back null, so that the getInputStream() throws a NPE. It would have been good to break it out, though for debugging purposes:

Regardless, the JavaDocs say that Runtime.exec() returns null only if the cmdArray or one if its elements is null. Security Manager issues throw unique security exceptions.

I'm strongly suspecting that the difference is that the process that builds cmdArray is not doing what you think it's doing and that while you're parked at a breakpoint on the "istream =" statement you should inspect cmdArray.

A valid cmdArray for Linux would have values like this:

I don't know the exact values that would work on your Windows system, but that's how things should look,



Right, I also print out the values of the cmdArray.  (Yes, that's how they look)

And, if I take them and use them in the Terminal it works.

The thing you keep overlooking is that this works fine if Tomcat is not running through the Tomcat Service (via Windows Services). The code and paths are OK. There's something else going on.


I'm not sure why this would make a difference, but all the code and paths are the same.

In Summary:

WORKS: Run Tomcat using "catalina start" from the "bin" folder of a separate Tomcat folder.

FAILS:   Run Tomcat using Windows Service Manager.

Same EXACT WAR file.

(The exact same WAR file has also always worked on the Mac, albeit with a different path.)

Baffled.

Thanks,
2 months ago

Tim Moores wrote:

Mike London wrote:This command equates to: "python <path_to_python_program> <params>"



My post in the other topic wrote:One possible gotcha is not to use absolute paths for everything - the executable as well as all files and/or directories that may be passed in as parameters.



Have you ensured that ALL paths and files are absolute, including the "python" command?



Thanks for your reply!

Yes, the paths are absolute and I have a method that checks the paths first just to be sure.

I just checked again and running the exact same code from a standalone Tomcat instance works, but the same WAR file running from the Tomcat Service manager does not.

Here is the error stack. I hope you can see something that helps!

Thank you!!!

-----

""2021-09-14 09:59:56 [http-nio-8080-exec-1] ERROR o.s.b.w.s.support.ErrorPageFilter - Forwarding to error page from request [/runPythonWithParam/] due to exception [null]
"java.lang.NullPointerException: null
at com.alias.ws.service.commandline.RunPythonService.runPythonServiceParam(RunPythonService.java:156)
at com.alias.ws.controllers.commandline.RunPythonController.runPythonParam(RunPythonController.java:192)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1064)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at com.alias.ws.controllers.filters.RequestResponseLoggingFilter.doFilter(RequestResponseLoggingFilter.java:43)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126)
at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64)
at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:836)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1839)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
2 months ago

Rob Spoor wrote:If you use ProcessBuilder and its redirectErrorStream method, you can redirect the error output to the standard output. That way you get one stream that contains both.



Good suggestion....but the stream is just the null followed by the unhelpful error stack (since I already know where the problem starts).

I also put in code to log the actual command being passed to System.getRuntime().exec().

This command equates to: "python <path_to_python_program> <params>"

That exact string (where I wrote it to a debug statement) works fine in Windows at command line but not in the java code.

I also checked to make sure the Java program could read the Python script by putting in debug statements into it (like file exists check, and such).

And, I put in code into the Python file to make sure parts of it at least were getting read by Java.

Here's the thing: none of the python file was read -- none of the Python file statements ran at all. Java (running under Tomcat Service) simply did not execute the Python program. At all.

--

Yet, again, if I start a standalone version of Tomcat (not run from the Windows Service manager) with the exact same WAR file, it all works fine.

I can reproduce this same problem on Windows 10 and Windows Server 2019.

I'm using JDK 11 (Amazon Corretto).

--

What could I still be missing?

Permissions issue of some kind?

The Python program is in a folder called C:\pythonscripts.

Thanks again.

-- mike

2 months ago