Skip to content

Instantly share code, notes, and snippets.

@zyazxr

zyazxr/gist:b8be22b7cdf48d0c60d6facd12515f20

Created January 2, 2020 04:18
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save zyazxr/b8be22b7cdf48d0c60d6facd12515f20 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save zyazxr/b8be22b7cdf48d0c60d6facd12515f20 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
"C:\Program Files\Java\jdk1.8.0_221\bin\java.exe" -XX:TieredStopAtLevel=1 -noverify -Dspring.output.ansi.enabled=always -Dcom.sun.management.jmxremote -Dspring.jmx.enabled=true -Dspring.liveBeansView.mbeanDomain -Dspring.application.admin.enabled=true "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2019.2.3\lib\idea_rt.jar=49396:C:\Program Files\JetBrains\IntelliJ IDEA 2019.2.3\bin" -Dfile.encoding=UTF-8 -classpath C:\Users\zhaoy\AppData\Local\Temp\classpath1215851767.jar com.nari.bdp.DnnSpringbootApplication
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/C:/Users/zhaoy/.m2/repository/ch/qos/logback/logback-classic/1.2.3/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/C:/Users/zhaoy/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:06.084][INFO ][main][DnnSpringbootApplication.java:91]-[traceId:-1577938206082][logFlag:SpringBoot2Application.getConfig] C:\Users\zhaoy\IdeaProjects\nari.work\work3\bdp_gitlab_ustc\bdp/conf/bdp.sys
env:PRO
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:06.114][INFO ][main][DnnSpringbootApplication.java:102]-[traceId:-1577938206114][logFlag:SpringBoot2Application.getConfig] PRO
apollo.meta:http://192.168.98.44:20551
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:06.116][INFO ][main][DnnSpringbootApplication.java:106]-[traceId:-1577938206116][logFlag:SpringBoot2Application.getConfig] http://192.168.98.44:20551
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:07.286][INFO ][main][DefaultApplicationProvider.java:93]-App ID is set to bdp-service-mine-dl-classification-dnn by app.id property from /META-INF/app.properties
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:07.292][INFO ][main][DefaultServerProvider.java:34]-Loading C:\opt\settings\server.properties
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:07.293][INFO ][main][DefaultServerProvider.java:108]-Environment is set to [PRO] by JVM system property 'env'.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:07.457][INFO ][main][DefaultMetaServerProvider.java:42]-Located meta services from apollo.meta configuration: http://192.168.98.44:20551!
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:07.468][WARN ][main][ResourceUtils.java:54]-No available properties: apollo-env.properties
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:07.474][INFO ][main][MetaDomainConsts.java:93]-Located meta server address http://192.168.98.44:20551 for env PRO from com.ctrip.framework.apollo.internals.DefaultMetaServerProvider
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:12.760][INFO ][main][DnnSpringbootApplication.java:127]-[traceId:-1577938212760[logFlag:SpringBoot2Application.getConfig}] From Apollo bdp.base.config namespace get eurekaUrl success, eurekaUrl: http://192.168.98.44:8761/eureka/
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:12.774][INFO ][main][DnnSpringbootApplication.java:136]-[traceId:-1577938212774[logFlag:SpringBoot2Application.getConfig}] From Apollo bdp.base.config namespace get server.log.RefreshBreaker success, refreshBreak: 3
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:12.787][INFO ][main][DnnSpringbootApplication.java:147]-[traceId:-1577938212787[logFlag:SpringBoot2Application.getConfig}] From Apollo get async.executor.thread.core_pool_size success: 5
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:12.788][INFO ][main][DnnSpringbootApplication.java:156]-[traceId:-1577938212788[logFlag:SpringBoot2Application.getConfig}] From Apollo get async.executor.thread.max_pool_size success: 10
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:12.788][INFO ][main][DnnSpringbootApplication.java:165]-[traceId:-1577938212788[logFlag:SpringBoot2Application.getConfig}] From Apollo get async.executor.thread.queue_capacity success: 100
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:12.788][INFO ][main][DnnSpringbootApplication.java:174]-[traceId:-1577938212788[logFlag:SpringBoot2Application.getConfig}] From Apollo get async.executor.thread.name.prefix success: thread-
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:14.332][INFO ][main][PostProcessorRegistrationDelegate.java:330]-Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$39426b80] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.3.RELEASE)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:14.790][INFO ][main][SpringApplication.java:675]-No active profile set, falling back to default profiles: default
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:16.735][INFO ][main][GenericScope.java:294]-BeanFactory id=a7da2908-41b5-3221-ab2c-b2751cc058b4
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:16.848][INFO ][main][PostProcessorRegistrationDelegate.java:330]-Bean 'com.nari.bdp.commons.mine.base.behaviors.hdfs.HDFSClient' of type [org.springframework.cloud.openfeign.FeignClientFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:16.852][INFO ][main][PostProcessorRegistrationDelegate.java:330]-Bean 'com.nari.bdp.commons.mine.base.behaviors.rdb.RDBClient' of type [org.springframework.cloud.openfeign.FeignClientFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:17.163][INFO ][main][PostProcessorRegistrationDelegate.java:330]-Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$39426b80] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:17.908][INFO ][main][TomcatWebServer.java:90]-Tomcat initialized with port(s): 20779 (http)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:17.939][INFO ][main][DirectJDKLog.java:173]-Initializing ProtocolHandler ["http-nio-20779"]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:17.954][INFO ][main][DirectJDKLog.java:173]-Starting service [Tomcat]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:17.955][INFO ][main][DirectJDKLog.java:173]-Starting Servlet engine: [Apache Tomcat/9.0.27]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:17.963][INFO ][main][DirectJDKLog.java:173]-The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [C:\Program Files\Java\jdk1.8.0_221\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\Java\jdk1.8.0_221\bin;C:\Program Files\JetBrains\IntelliJ IDEA 2019.2.3\plugins\maven\lib\maven3\bin;C:\Program Files\hadoop-3.0.0\bin;C:\Program Files (x86)\scala\bin;C:\Users\zhaoy\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Bandizip\;.]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:18.124][INFO ][main][DirectJDKLog.java:173]-Initializing Spring embedded WebApplicationContext
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:18.125][INFO ][main][ServletWebServerApplicationContext.java:296]-Root WebApplicationContext: initialization completed in 3299 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:19.364][INFO ][main][PropertySourcedRequestMappingHandlerMapping.java:69]-Mapped URL path [/v2/api-docs] onto method [public org.springframework.http.ResponseEntity<springfox.documentation.spring.web.json.Json> springfox.documentation.swagger2.web.Swagger2Controller.getDocumentation(java.lang.String,javax.servlet.http.HttpServletRequest)]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:19.509][WARN ][main][URLConfigurationSource.java:121]-No URLs will be polled as dynamic configuration sources.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:19.509][INFO ][main][URLConfigurationSource.java:122]-To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:19.518][WARN ][main][URLConfigurationSource.java:121]-No URLs will be polled as dynamic configuration sources.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:19.519][INFO ][main][URLConfigurationSource.java:122]-To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:20.072][INFO ][main][ExecutorConfigurationSupport.java:171]-Initializing ExecutorService 'applicationTaskExecutor'
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:21.604][INFO ][main][InstanceInfoFactory.java:71]-Setting initial instance status as: STARTING
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:21.679][INFO ][main][DiscoveryClient.java:349]-Initializing Eureka in region us-east-1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:22.596][INFO ][main][DiscoveryJerseyProvider.java:70]-Using JSON encoding codec LegacyJacksonJson
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:22.602][INFO ][main][DiscoveryJerseyProvider.java:71]-Using JSON decoding codec LegacyJacksonJson
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:23.430][INFO ][main][DiscoveryJerseyProvider.java:80]-Using XML encoding codec XStreamXml
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:23.432][INFO ][main][DiscoveryJerseyProvider.java:81]-Using XML decoding codec XStreamXml
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:24.795][INFO ][main][ConfigClusterResolver.java:43]-Resolving eureka endpoints via configuration
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.058][INFO ][main][DiscoveryClient.java:958]-Disable delta property : false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.059][INFO ][main][DiscoveryClient.java:959]-Single vip registry refresh property : null
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.059][INFO ][main][DiscoveryClient.java:960]-Force full registry fetch : false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.060][INFO ][main][DiscoveryClient.java:961]-Application is null : false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.061][INFO ][main][DiscoveryClient.java:962]-Registered Applications size is zero : true
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.061][INFO ][main][DiscoveryClient.java:964]-Application version is -1: true
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.062][INFO ][main][DiscoveryClient.java:1047]-Getting all instance registry info from the eureka server
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.600][INFO ][main][DiscoveryClient.java:1056]-The response status is 200
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.605][INFO ][main][DiscoveryClient.java:1270]-Starting heartbeat executor: renew interval is: 30
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.608][INFO ][main][InstanceInfoReplicator.java:60]-InstanceInfoReplicator onDemand update allowed rate per min is 4
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.631][INFO ][main][DiscoveryClient.java:449]-Discovery Client initialized at timestamp 1577938225629 with initial instances count: 26
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.649][INFO ][main][EurekaServiceRegistry.java:42]-Registering application BDP-SERVICE-MINE-DL-CLASSIFICATION-DNN with eureka with status UP
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.651][INFO ][main][DiscoveryClient.java:1305]-Saw local status change event StatusChangeEvent [timestamp=1577938225650, current=UP, previous=STARTING]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.653][INFO ][main][DocumentationPluginsBootstrapper.java:151]-Context refreshed
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.654][INFO ][DiscoveryClient-InstanceInfoReplicator-0][DiscoveryClient.java:826]-DiscoveryClient_BDP-SERVICE-MINE-DL-CLASSIFICATION-DNN/DESKTOP-ZY:bdp-service-mine-dl-classification-dnn:20779: registering service...
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.674][INFO ][main][DocumentationPluginsBootstrapper.java:154]-Found 1 custom documentation plugin(s)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.721][INFO ][DiscoveryClient-InstanceInfoReplicator-0][DiscoveryClient.java:835]-DiscoveryClient_BDP-SERVICE-MINE-DL-CLASSIFICATION-DNN/DESKTOP-ZY:bdp-service-mine-dl-classification-dnn:20779 - registration status: 204
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:25.795][INFO ][main][ApiListingReferenceScanner.java:41]-Scanning for api listing references
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:26.217][INFO ][main][DirectJDKLog.java:173]-Starting ProtocolHandler ["http-nio-20779"]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:26.282][INFO ][main][TomcatWebServer.java:204]-Tomcat started on port(s): 20779 (http) with context path ''
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:26.283][INFO ][main][EurekaAutoServiceRegistration.java:141]-Updating port to 20779
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:26.286][INFO ][main][StartupInfoLogger.java:59]-Started DnnSpringbootApplication in 13.377 seconds (JVM running for 22.671)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:26.289][INFO ][main][DnnSpringbootApplication.java:189]-[traceId:-1577938226289][logFlag:DnnSpringbootApplication.main] Application started successfully.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:30.192][INFO ][http-nio-20779-exec-1][DirectJDKLog.java:173]-Initializing Spring DispatcherServlet 'dispatcherServlet'
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:30.192][INFO ][http-nio-20779-exec-1][FrameworkServlet.java:524]-Initializing Servlet 'dispatcherServlet'
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:30.218][INFO ][http-nio-20779-exec-1][FrameworkServlet.java:546]-Completed initialization in 25 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.281][INFO ][http-nio-20779-exec-1][ChainedDynamicProperty.java:115]-Flipping property: bdp-service-base-data-rdb.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.353][INFO ][http-nio-20779-exec-1][ShutdownEnabledTimer.java:58]-Shutdown hook installed for: NFLoadBalancer-PingTimer-bdp-service-base-data-rdb
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.354][INFO ][http-nio-20779-exec-1][BaseLoadBalancer.java:197]-Client: bdp-service-base-data-rdb instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=bdp-service-base-data-rdb,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.369][INFO ][http-nio-20779-exec-1][DynamicServerListLoadBalancer.java:222]-Using serverListUpdater PollingServerListUpdater
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.400][INFO ][http-nio-20779-exec-1][ChainedDynamicProperty.java:115]-Flipping property: bdp-service-base-data-rdb.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.403][INFO ][http-nio-20779-exec-1][DynamicServerListLoadBalancer.java:150]-DynamicServerListLoadBalancer for client bdp-service-base-data-rdb initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=bdp-service-base-data-rdb,current list of Servers=[nrdb2:20402, nrdb1:20402],Load balancer stats=Zone stats: {defaultzone=[Zone:defaultzone; Instance count:2; Active connections count: 0; Circuit breaker tripped count: 0; Active connections per server: 0.0;]
},Server stats: [[Server:nrdb2:20402; Zone:defaultZone; Total Requests:0; Successive connection failure:0; Total blackout seconds:0; Last connection made:Thu Jan 01 08:00:00 CST 1970; First connection made: Thu Jan 01 08:00:00 CST 1970; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp time:0.0; stddev resp time:0.0]
, [Server:nrdb1:20402; Zone:defaultZone; Total Requests:0; Successive connection failure:0; Total blackout seconds:0; Last connection made:Thu Jan 01 08:00:00 CST 1970; First connection made: Thu Jan 01 08:00:00 CST 1970; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp time:0.0; stddev resp time:0.0]
]}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList@751b27ef
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.971][INFO ][http-nio-20779-exec-1][ChainedDynamicProperty.java:115]-Flipping property: bdp-service-base-data-hdfs.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.975][INFO ][http-nio-20779-exec-1][ShutdownEnabledTimer.java:58]-Shutdown hook installed for: NFLoadBalancer-PingTimer-bdp-service-base-data-hdfs
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.976][INFO ][http-nio-20779-exec-1][BaseLoadBalancer.java:197]-Client: bdp-service-base-data-hdfs instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=bdp-service-base-data-hdfs,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.991][INFO ][http-nio-20779-exec-1][DynamicServerListLoadBalancer.java:222]-Using serverListUpdater PollingServerListUpdater
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.995][INFO ][http-nio-20779-exec-1][ChainedDynamicProperty.java:115]-Flipping property: bdp-service-base-data-hdfs.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:31.997][INFO ][http-nio-20779-exec-1][DynamicServerListLoadBalancer.java:150]-DynamicServerListLoadBalancer for client bdp-service-base-data-hdfs initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=bdp-service-base-data-hdfs,current list of Servers=[nrdb1:20403, nrdb2:20403],Load balancer stats=Zone stats: {defaultzone=[Zone:defaultzone; Instance count:2; Active connections count: 0; Circuit breaker tripped count: 0; Active connections per server: 0.0;]
},Server stats: [[Server:nrdb2:20403; Zone:defaultZone; Total Requests:0; Successive connection failure:0; Total blackout seconds:0; Last connection made:Thu Jan 01 08:00:00 CST 1970; First connection made: Thu Jan 01 08:00:00 CST 1970; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp time:0.0; stddev resp time:0.0]
, [Server:nrdb1:20403; Zone:defaultZone; Total Requests:0; Successive connection failure:0; Total blackout seconds:0; Last connection made:Thu Jan 01 08:00:00 CST 1970; First connection made: Thu Jan 01 08:00:00 CST 1970; Active Connections:0; total failure count in last (1000) msecs:0; average resp time:0.0; 90 percentile resp time:0.0; 95 percentile resp time:0.0; min resp time:0.0; max resp time:0.0; stddev resp time:0.0]
]}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList@26ef7d10
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:32.461][INFO ][PollingServerListUpdater-0][ChainedDynamicProperty.java:115]-Flipping property: bdp-service-base-data-rdb.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:33.003][INFO ][PollingServerListUpdater-1][ChainedDynamicProperty.java:115]-Flipping property: bdp-service-base-data-hdfs.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:37.576][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Running Spark version 2.4.0.cloudera2
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:37.883][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Submitted application: job
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:37.959][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Changing view acls to: zhaoy
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:37.960][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Changing modify acls to: zhaoy
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:37.962][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Changing view acls groups to:
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:37.963][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Changing modify acls groups to:
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:37.965][INFO ][http-nio-20779-exec-1][Logging.scala:54]-SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(zhaoy); groups with view permissions: Set(); users with modify permissions: Set(zhaoy); groups with modify permissions: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:38.222][INFO ][http-nio-20779-exec-1][Logging.scala:54]-max retries is 16
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:38.682][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Successfully started service 'sparkDriver' on port 49441.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:38.711][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Registering MapOutputTracker
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:38.732][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Registering BlockManagerMaster
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:38.737][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:38.738][INFO ][http-nio-20779-exec-1][Logging.scala:54]-BlockManagerMasterEndpoint up
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:38.746][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Created local directory at C:\Users\zhaoy\AppData\Local\Temp\blockmgr-a5dd7ff4-45a1-4f81-8e73-c28f87973f0e
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:38.791][INFO ][http-nio-20779-exec-1][Logging.scala:54]-MemoryStore started with capacity 1985.4 MB
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:38.813][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Registering OutputCommitCoordinator
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:38.910][INFO ][http-nio-20779-exec-1][Log.java:192]-Logging initialized @35295ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.045][INFO ][http-nio-20779-exec-1][Server.java:351]-jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.064][INFO ][http-nio-20779-exec-1][Server.java:419]-Started @35449ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.068][INFO ][http-nio-20779-exec-1][Logging.scala:54]-max retries is 16
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.093][INFO ][http-nio-20779-exec-1][AbstractConnector.java:278]-Started ServerConnector@5c1a1346{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.093][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Successfully started service 'SparkUI' on port 4040.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.120][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@12b8cb6b{/jobs,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.121][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@2cb0e22e{/jobs/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.123][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@749528d1{/jobs/job,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.125][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@20cd0173{/jobs/job/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.127][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@50707b11{/stages,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.129][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@ba4a897{/stages/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.131][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@c14ffb7{/stages/stage,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.138][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@4f24ff68{/stages/stage/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.140][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@5eb9f733{/stages/pool,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.142][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@2f4be922{/stages/pool/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.143][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@2c2b53ba{/storage,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.145][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@464a766c{/storage/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.147][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@315c4{/storage/rdd,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.148][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@60a3596e{/storage/rdd/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.150][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@7010d27a{/environment,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.153][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@7b675994{/environment/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.155][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@1d0632b8{/executors,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.157][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@639c895{/executors/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.159][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@2a4c0ca7{/executors/threadDump,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.161][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@6ea566b2{/executors/threadDump/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.165][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@701aba71{/static,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.171][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@1f669f09{/,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.175][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@2dcbc56b{/api,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.177][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@7b320213{/jobs/job/kill,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.179][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@6e33e659{/stages/stage/kill,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.184][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Bound SparkUI to 0.0.0.0, and started at http://DESKTOP-ZY:4040
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.424][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting executor ID driver on host localhost
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.463][INFO ][http-nio-20779-exec-1][Logging.scala:54]-max retries is 16
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.467][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 49462.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.469][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Server created on DESKTOP-ZY:49462
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.471][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.524][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Registering BlockManager BlockManagerId(driver, DESKTOP-ZY, 49462, None)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.529][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Registering block manager DESKTOP-ZY:49462 with 1985.4 MB RAM, BlockManagerId(driver, DESKTOP-ZY, 49462, None)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.534][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Registered BlockManager BlockManagerId(driver, DESKTOP-ZY, 49462, None)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.535][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Initialized BlockManager: BlockManagerId(driver, DESKTOP-ZY, 49462, None)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:39.560][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@243d58d8{/metrics/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:40.627][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('C:\Users\zhaoy\AppData\Local\Temp\').
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:40.628][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Warehouse path is 'C:\Users\zhaoy\AppData\Local\Temp\'.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:40.639][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@28ff5d5{/SQL,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:40.641][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@4cfd2cd5{/SQL/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:40.643][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@7bfa2da3{/SQL/execution,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:40.645][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@2e69f26e{/SQL/execution/json,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:40.647][INFO ][http-nio-20779-exec-1][ContextHandler.java:781]-Started o.s.j.s.ServletContextHandler@68ac3c10{/static/sql,null,AVAILABLE,@Spark}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:41.581][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Registered StateStoreCoordinator endpoint
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:51.041][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 1237.9616 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:51.125][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 24.5197 ms
+---+---+------+----------+---------+
|150| 4|setosa|versicolor|virginica|
+---+---+------+----------+---------+
|5.1|3.5| 1.4| 0.2| ww0|
|4.9|3.0| 1.4| 0.2| ww0|
|4.7|3.2| 1.3| 0.2| ww0|
|4.6|3.1| 1.5| 0.2| ww0|
|5.0|3.6| 1.4| 0.2| ww0|
|5.4|3.9| 1.7| 0.4| ww0|
|4.6|3.4| 1.4| 0.3| ww0|
|5.0|3.4| 1.5| 0.2| ww0|
|4.4|2.9| 1.4| 0.2| ww0|
|4.9|3.1| 1.5| 0.1| ww0|
|5.4|3.7| 1.5| 0.2| ww0|
|4.8|3.4| 1.6| 0.2| ww0|
|4.8|3.0| 1.4| 0.1| ww0|
|4.3|3.0| 1.1| 0.1| ww0|
|5.8|4.0| 1.2| 0.2| ww0|
|5.7|4.4| 1.5| 0.4| ww0|
|5.4|3.9| 1.3| 0.4| ww0|
|5.1|3.5| 1.4| 0.3| ww0|
|5.7|3.8| 1.7| 0.3| ww0|
|5.1|3.8| 1.5| 0.3| ww0|
+---+---+------+----------+---------+
only showing top 20 rows
+---+---+------+----------+---------+
|150| 4|setosa|versicolor|virginica|
+---+---+------+----------+---------+
|5.1|3.5| 1.4| 0.2| ww0|
|4.9|3.0| 1.4| 0.2| ww0|
|4.7|3.2| 1.3| 0.2| ww0|
|4.6|3.1| 1.5| 0.2| ww0|
|5.0|3.6| 1.4| 0.2| ww0|
|5.4|3.9| 1.7| 0.4| ww0|
|4.6|3.4| 1.4| 0.3| ww0|
|5.0|3.4| 1.5| 0.2| ww0|
|4.4|2.9| 1.4| 0.2| ww0|
|4.9|3.1| 1.5| 0.1| ww0|
|5.4|3.7| 1.5| 0.2| ww0|
|4.8|3.4| 1.6| 0.2| ww0|
|4.8|3.0| 1.4| 0.1| ww0|
|4.3|3.0| 1.1| 0.1| ww0|
|5.8|4.0| 1.2| 0.2| ww0|
|5.7|4.4| 1.5| 0.4| ww0|
|5.4|3.9| 1.3| 0.4| ww0|
|5.1|3.5| 1.4| 0.3| ww0|
|5.7|3.8| 1.7| 0.3| ww0|
|5.1|3.8| 1.5| 0.3| ww0|
+---+---+------+----------+---------+
only showing top 20 rows
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.006][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 42.3163 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.043][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 27.657 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.057][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 11.0479 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.378][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: count at MineBaseUtil.java:251
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.419][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 2
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.420][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.420][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 0
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.457][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Registering RDD 3 (count at MineBaseUtil.java:251)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.472][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 0 (count at MineBaseUtil.java:251) with 1 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.473][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 1 (count at MineBaseUtil.java:251)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.474][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List(ShuffleMapStage 0)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.487][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List(ShuffleMapStage 0)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.510][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ShuffleMapStage 0 (MapPartitionsRDD[3] at count at MineBaseUtil.java:251), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.720][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_0 stored as values in memory (estimated size 7.9 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.875][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_0_piece0 stored as bytes in memory (estimated size 4.2 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.881][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Added broadcast_0_piece0 in memory on DESKTOP-ZY:49462 (size: 4.2 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.893][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 0 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.906][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at count at MineBaseUtil.java:251) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:52.909][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 0.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.049][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 8629 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.053][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 1.0 in stage 0.0 (TID 1, localhost, executor driver, partition 1, PROCESS_LOCAL, 8646 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.054][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 2.0 in stage 0.0 (TID 2, localhost, executor driver, partition 2, PROCESS_LOCAL, 8629 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.379][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 3.0 in stage 0.0 (TID 3, localhost, executor driver, partition 3, PROCESS_LOCAL, 8646 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.400][INFO ][Executor task launch worker for task 2][Logging.scala:54]-Running task 2.0 in stage 0.0 (TID 2)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.400][INFO ][Executor task launch worker for task 0][Logging.scala:54]-Running task 0.0 in stage 0.0 (TID 0)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.400][INFO ][Executor task launch worker for task 1][Logging.scala:54]-Running task 1.0 in stage 0.0 (TID 1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.400][INFO ][Executor task launch worker for task 3][Logging.scala:54]-Running task 3.0 in stage 0.0 (TID 3)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.634][INFO ][Executor task launch worker for task 3][Logging.scala:54]-Finished task 3.0 in stage 0.0 (TID 3). 1543 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.634][INFO ][Executor task launch worker for task 0][Logging.scala:54]-Finished task 0.0 in stage 0.0 (TID 0). 1543 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.634][INFO ][Executor task launch worker for task 2][Logging.scala:54]-Finished task 2.0 in stage 0.0 (TID 2). 1543 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.634][INFO ][Executor task launch worker for task 1][Logging.scala:54]-Finished task 1.0 in stage 0.0 (TID 1). 1500 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.663][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 1.0 in stage 0.0 (TID 1) in 607 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.669][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 2.0 in stage 0.0 (TID 2) in 616 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.670][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 3.0 in stage 0.0 (TID 3) in 292 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.670][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 0.0 in stage 0.0 (TID 0) in 689 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.672][INFO ][task-result-getter-1][Logging.scala:54]-Removed TaskSet 0.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.687][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ShuffleMapStage 0 (count at MineBaseUtil.java:251) finished in 1.128 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.688][INFO ][dag-scheduler-event-loop][Logging.scala:54]-looking for newly runnable stages
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.689][INFO ][dag-scheduler-event-loop][Logging.scala:54]-running: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.690][INFO ][dag-scheduler-event-loop][Logging.scala:54]-waiting: Set(ResultStage 1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.691][INFO ][dag-scheduler-event-loop][Logging.scala:54]-failed: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.697][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 1 (MapPartitionsRDD[6] at count at MineBaseUtil.java:251), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.714][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_1 stored as values in memory (estimated size 7.1 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.716][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_1_piece0 stored as bytes in memory (estimated size 3.8 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.718][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Added broadcast_1_piece0 in memory on DESKTOP-ZY:49462 (size: 3.8 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.719][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 1 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.721][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[6] at count at MineBaseUtil.java:251) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.721][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 1.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.731][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 0.0 in stage 1.0 (TID 4, localhost, executor driver, partition 0, ANY, 7767 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.731][INFO ][Executor task launch worker for task 4][Logging.scala:54]-Running task 0.0 in stage 1.0 (TID 4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.755][INFO ][Executor task launch worker for task 4][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.759][INFO ][Executor task launch worker for task 4][Logging.scala:54]-Started 0 remote fetches in 12 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.797][INFO ][Executor task launch worker for task 4][Logging.scala:54]-Finished task 0.0 in stage 1.0 (TID 4). 1739 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.798][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 0.0 in stage 1.0 (TID 4) in 70 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.798][INFO ][task-result-getter-3][Logging.scala:54]-Removed TaskSet 1.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.799][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 1 (count at MineBaseUtil.java:251) finished in 0.090 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:53.806][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 0 finished: count at MineBaseUtil.java:251, took 1.425218 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:54.336][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 19.5886 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:54.521][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: countByValue at StringIndexer.scala:140
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.282][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Registering RDD 13 (countByValue at StringIndexer.scala:140)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.282][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 1 (countByValue at StringIndexer.scala:140) with 4 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.282][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 3 (countByValue at StringIndexer.scala:140)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.282][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List(ShuffleMapStage 2)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.282][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List(ShuffleMapStage 2)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.283][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ShuffleMapStage 2 (MapPartitionsRDD[13] at countByValue at StringIndexer.scala:140), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.307][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_2 stored as values in memory (estimated size 7.6 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.313][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_2_piece0 stored as bytes in memory (estimated size 4.2 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.316][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Added broadcast_2_piece0 in memory on DESKTOP-ZY:49462 (size: 4.2 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.317][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 2 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.319][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[13] at countByValue at StringIndexer.scala:140) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.319][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 2.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.320][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 0.0 in stage 2.0 (TID 5, localhost, executor driver, partition 0, PROCESS_LOCAL, 9517 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.321][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 1.0 in stage 2.0 (TID 6, localhost, executor driver, partition 1, PROCESS_LOCAL, 9558 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.321][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 2.0 in stage 2.0 (TID 7, localhost, executor driver, partition 2, PROCESS_LOCAL, 9517 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.322][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 3.0 in stage 2.0 (TID 8, localhost, executor driver, partition 3, PROCESS_LOCAL, 9558 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.322][INFO ][Executor task launch worker for task 5][Logging.scala:54]-Running task 0.0 in stage 2.0 (TID 5)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.323][INFO ][Executor task launch worker for task 6][Logging.scala:54]-Running task 1.0 in stage 2.0 (TID 6)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.323][INFO ][Executor task launch worker for task 7][Logging.scala:54]-Running task 2.0 in stage 2.0 (TID 7)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.323][INFO ][Executor task launch worker for task 8][Logging.scala:54]-Running task 3.0 in stage 2.0 (TID 8)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.352][INFO ][Executor task launch worker for task 6][Logging.scala:54]-Code generated in 18.2965 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.458][INFO ][Executor task launch worker for task 7][Logging.scala:54]-Finished task 2.0 in stage 2.0 (TID 7). 1244 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.460][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 2.0 in stage 2.0 (TID 7) in 139 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.469][INFO ][Executor task launch worker for task 5][Logging.scala:54]-Finished task 0.0 in stage 2.0 (TID 5). 1244 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.470][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 0.0 in stage 2.0 (TID 5) in 150 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.480][INFO ][Executor task launch worker for task 6][Logging.scala:54]-Finished task 1.0 in stage 2.0 (TID 6). 1244 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.482][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 1.0 in stage 2.0 (TID 6) in 161 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.490][INFO ][Executor task launch worker for task 8][Logging.scala:54]-Finished task 3.0 in stage 2.0 (TID 8). 1244 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.492][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 3.0 in stage 2.0 (TID 8) in 170 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.492][INFO ][task-result-getter-3][Logging.scala:54]-Removed TaskSet 2.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.493][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ShuffleMapStage 2 (countByValue at StringIndexer.scala:140) finished in 0.209 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.494][INFO ][dag-scheduler-event-loop][Logging.scala:54]-looking for newly runnable stages
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.494][INFO ][dag-scheduler-event-loop][Logging.scala:54]-running: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.494][INFO ][dag-scheduler-event-loop][Logging.scala:54]-waiting: Set(ResultStage 3)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.494][INFO ][dag-scheduler-event-loop][Logging.scala:54]-failed: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.494][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 3 (ShuffledRDD[14] at countByValue at StringIndexer.scala:140), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.497][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_3 stored as values in memory (estimated size 3.3 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.502][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_3_piece0 stored as bytes in memory (estimated size 2014.0 B, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.503][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Added broadcast_3_piece0 in memory on DESKTOP-ZY:49462 (size: 2014.0 B, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.504][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 3 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.504][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ResultStage 3 (ShuffledRDD[14] at countByValue at StringIndexer.scala:140) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.505][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 3.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.508][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Starting task 3.0 in stage 3.0 (TID 9, localhost, executor driver, partition 3, PROCESS_LOCAL, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.509][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Starting task 0.0 in stage 3.0 (TID 10, localhost, executor driver, partition 0, ANY, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.510][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Starting task 1.0 in stage 3.0 (TID 11, localhost, executor driver, partition 1, ANY, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.510][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Starting task 2.0 in stage 3.0 (TID 12, localhost, executor driver, partition 2, ANY, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.510][INFO ][Executor task launch worker for task 9][Logging.scala:54]-Running task 3.0 in stage 3.0 (TID 9)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.510][INFO ][Executor task launch worker for task 10][Logging.scala:54]-Running task 0.0 in stage 3.0 (TID 10)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.512][INFO ][Executor task launch worker for task 11][Logging.scala:54]-Running task 1.0 in stage 3.0 (TID 11)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.513][INFO ][Executor task launch worker for task 9][Logging.scala:54]-Getting 0 non-empty blocks including 0 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.515][INFO ][Executor task launch worker for task 9][Logging.scala:54]-Started 0 remote fetches in 2 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.516][INFO ][Executor task launch worker for task 11][Logging.scala:54]-Getting 2 non-empty blocks including 2 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.516][INFO ][Executor task launch worker for task 11][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.519][INFO ][Executor task launch worker for task 10][Logging.scala:54]-Getting 2 non-empty blocks including 2 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.519][INFO ][Executor task launch worker for task 10][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.529][INFO ][Executor task launch worker for task 9][Logging.scala:54]-Finished task 3.0 in stage 3.0 (TID 9). 1091 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.530][INFO ][Executor task launch worker for task 11][Logging.scala:54]-Finished task 1.0 in stage 3.0 (TID 11). 1240 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.531][INFO ][Executor task launch worker for task 10][Logging.scala:54]-Finished task 0.0 in stage 3.0 (TID 10). 1197 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.532][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 1.0 in stage 3.0 (TID 11) in 23 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.532][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 0.0 in stage 3.0 (TID 10) in 23 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.533][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 3.0 in stage 3.0 (TID 9) in 27 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.535][INFO ][Executor task launch worker for task 12][Logging.scala:54]-Running task 2.0 in stage 3.0 (TID 12)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.537][INFO ][Executor task launch worker for task 12][Logging.scala:54]-Getting 2 non-empty blocks including 2 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.537][INFO ][Executor task launch worker for task 12][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.550][INFO ][Executor task launch worker for task 12][Logging.scala:54]-Finished task 2.0 in stage 3.0 (TID 12). 1240 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.552][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 2.0 in stage 3.0 (TID 12) in 42 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.553][INFO ][task-result-getter-3][Logging.scala:54]-Removed TaskSet 3.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.554][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 3 (countByValue at StringIndexer.scala:140) finished in 0.058 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.558][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 1 finished: countByValue at StringIndexer.scala:140, took 1.035973 s
+---+---+------+----------+---------+
|150| 4|setosa|versicolor|virginica|
+---+---+------+----------+---------+
|5.1|3.5| 1.4| 0.2| ww0|
|4.9|3.0| 1.4| 0.2| ww0|
|4.7|3.2| 1.3| 0.2| ww0|
|4.6|3.1| 1.5| 0.2| ww0|
|5.0|3.6| 1.4| 0.2| ww0|
|5.4|3.9| 1.7| 0.4| ww0|
|4.6|3.4| 1.4| 0.3| ww0|
|5.0|3.4| 1.5| 0.2| ww0|
|4.4|2.9| 1.4| 0.2| ww0|
|4.9|3.1| 1.5| 0.1| ww0|
|5.4|3.7| 1.5| 0.2| ww0|
|4.8|3.4| 1.6| 0.2| ww0|
|4.8|3.0| 1.4| 0.1| ww0|
|4.3|3.0| 1.1| 0.1| ww0|
|5.8|4.0| 1.2| 0.2| ww0|
|5.7|4.4| 1.5| 0.4| ww0|
|5.4|3.9| 1.3| 0.4| ww0|
|5.1|3.5| 1.4| 0.3| ww0|
|5.7|3.8| 1.7| 0.3| ww0|
|5.1|3.8| 1.5| 0.3| ww0|
|5.4|3.4| 1.7| 0.2| ww0|
|5.1|3.7| 1.5| 0.4| ww0|
|4.6|3.6| 1.0| 0.2| ww0|
|5.1|3.3| 1.7| 0.5| ww0|
|4.8|3.4| 1.9| 0.2| ww0|
|5.0|3.0| 1.6| 0.2| ww0|
|5.0|3.4| 1.6| 0.4| ww0|
|5.2|3.5| 1.5| 0.2| ww0|
|5.2|3.4| 1.4| 0.2| ww0|
|4.7|3.2| 1.6| 0.2| ww0|
|4.8|3.1| 1.6| 0.2| ww0|
|5.4|3.4| 1.5| 0.4| ww0|
|5.2|4.1| 1.5| 0.1| ww0|
|5.5|4.2| 1.4| 0.2| ww0|
|4.9|3.1| 1.5| 0.2| ww0|
|5.0|3.2| 1.2| 0.2| ww0|
|5.5|3.5| 1.3| 0.2| ww0|
|4.9|3.6| 1.4| 0.1| ww0|
|4.4|3.0| 1.3| 0.2| ww0|
|5.1|3.4| 1.5| 0.2| ww0|
|5.0|3.5| 1.3| 0.3| ww0|
|4.5|2.3| 1.3| 0.3| ww0|
|4.4|3.2| 1.3| 0.2| ww0|
|5.0|3.5| 1.6| 0.6| ww0|
|5.1|3.8| 1.9| 0.4| ww0|
|4.8|3.0| 1.4| 0.3| ww0|
|5.1|3.8| 1.6| 0.2| ww0|
|4.6|3.2| 1.4| 0.2| ww0|
|5.3|3.7| 1.5| 0.2| ww0|
|5.0|3.3| 1.4| 0.2| ww0|
|7.0|3.2| 4.7| 1.4| ww1|
|6.4|3.2| 4.5| 1.5| ww1|
|6.9|3.1| 4.9| 1.5| ww1|
|5.5|2.3| 4.0| 1.3| ww1|
|6.5|2.8| 4.6| 1.5| ww1|
|5.7|2.8| 4.5| 1.3| ww1|
|6.3|3.3| 4.7| 1.6| ww1|
|4.9|2.4| 3.3| 1.0| ww1|
|6.6|2.9| 4.6| 1.3| ww1|
|5.2|2.7| 3.9| 1.4| ww1|
|5.0|2.0| 3.5| 1.0| ww1|
|5.9|3.0| 4.2| 1.5| ww1|
|6.0|2.2| 4.0| 1.0| ww1|
|6.1|2.9| 4.7| 1.4| ww1|
|5.6|2.9| 3.6| 1.3| ww1|
|6.7|3.1| 4.4| 1.4| ww1|
|5.6|3.0| 4.5| 1.5| ww1|
|5.8|2.7| 4.1| 1.0| ww1|
|6.2|2.2| 4.5| 1.5| ww1|
|5.6|2.5| 3.9| 1.1| ww1|
|5.9|3.2| 4.8| 1.8| ww1|
|6.1|2.8| 4.0| 1.3| ww1|
|6.3|2.5| 4.9| 1.5| ww1|
|6.1|2.8| 4.7| 1.2| ww1|
|6.4|2.9| 4.3| 1.3| ww1|
|6.6|3.0| 4.4| 1.4| ww1|
|6.8|2.8| 4.8| 1.4| ww1|
|6.7|3.0| 5.0| 1.7| ww1|
|6.0|2.9| 4.5| 1.5| ww1|
|5.7|2.6| 3.5| 1.0| ww1|
|5.5|2.4| 3.8| 1.1| ww1|
|5.5|2.4| 3.7| 1.0| ww1|
|5.8|2.7| 3.9| 1.2| ww1|
|6.0|2.7| 5.1| 1.6| ww1|
|5.4|3.0| 4.5| 1.5| ww1|
|6.0|3.4| 4.5| 1.6| ww1|
|6.7|3.1| 4.7| 1.5| ww1|
|6.3|2.3| 4.4| 1.3| ww1|
|5.6|3.0| 4.1| 1.3| ww1|
|5.5|2.5| 4.0| 1.3| ww1|
|5.5|2.6| 4.4| 1.2| ww1|
|6.1|3.0| 4.6| 1.4| ww1|
|5.8|2.6| 4.0| 1.2| ww1|
|5.0|2.3| 3.3| 1.0| ww1|
|5.6|2.7| 4.2| 1.3| ww1|
|5.7|3.0| 4.2| 1.2| ww1|
|5.7|2.9| 4.2| 1.3| ww1|
|6.2|2.9| 4.3| 1.3| ww1|
|5.1|2.5| 3.0| 1.1| ww1|
|5.7|2.8| 4.1| 1.3| ww1|
+---+---+------+----------+---------+
only showing top 100 rows
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:55.986][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 19.4501 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:56.009][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 15.7504 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.192][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 32.8643 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.517][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: treeAggregate at StandardScaler.scala:57
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.518][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 2 (treeAggregate at StandardScaler.scala:57) with 4 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.518][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 4 (treeAggregate at StandardScaler.scala:57)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.519][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.519][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.520][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 4 (MapPartitionsRDD[20] at treeAggregate at StandardScaler.scala:57), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.525][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_4 stored as values in memory (estimated size 8.5 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.531][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_4_piece0 stored as bytes in memory (estimated size 4.6 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.532][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Added broadcast_4_piece0 in memory on DESKTOP-ZY:49462 (size: 4.6 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.532][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 4 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.533][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ResultStage 4 (MapPartitionsRDD[20] at treeAggregate at StandardScaler.scala:57) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.536][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 4.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.540][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 0.0 in stage 4.0 (TID 13, localhost, executor driver, partition 0, PROCESS_LOCAL, 12488 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.542][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 1.0 in stage 4.0 (TID 14, localhost, executor driver, partition 1, PROCESS_LOCAL, 12609 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.543][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 2.0 in stage 4.0 (TID 15, localhost, executor driver, partition 2, PROCESS_LOCAL, 12488 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.543][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 3.0 in stage 4.0 (TID 16, localhost, executor driver, partition 3, PROCESS_LOCAL, 12609 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.545][INFO ][Executor task launch worker for task 13][Logging.scala:54]-Running task 0.0 in stage 4.0 (TID 13)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.545][INFO ][Executor task launch worker for task 14][Logging.scala:54]-Running task 1.0 in stage 4.0 (TID 14)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.546][INFO ][Executor task launch worker for task 15][Logging.scala:54]-Running task 2.0 in stage 4.0 (TID 15)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.547][INFO ][Executor task launch worker for task 16][Logging.scala:54]-Running task 3.0 in stage 4.0 (TID 16)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.605][INFO ][Executor task launch worker for task 16][Logging.scala:54]-Code generated in 26.2941 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.656][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 95
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.683][INFO ][Executor task launch worker for task 13][Logging.scala:54]-Finished task 0.0 in stage 4.0 (TID 13). 1664 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.685][INFO ][Executor task launch worker for task 14][Logging.scala:54]-Finished task 1.0 in stage 4.0 (TID 14). 1621 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.687][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned shuffle 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.689][INFO ][Executor task launch worker for task 16][Logging.scala:54]-Finished task 3.0 in stage 4.0 (TID 16). 1621 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.691][INFO ][Executor task launch worker for task 15][Logging.scala:54]-Finished task 2.0 in stage 4.0 (TID 15). 1621 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.782][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 72
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.782][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 79
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.782][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 74
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.783][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 1.0 in stage 4.0 (TID 14) in 242 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.786][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 2.0 in stage 4.0 (TID 15) in 244 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.789][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 0.0 in stage 4.0 (TID 13) in 250 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.790][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 3.0 in stage 4.0 (TID 16) in 247 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.790][INFO ][task-result-getter-0][Logging.scala:54]-Removed TaskSet 4.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.793][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 4 (treeAggregate at StandardScaler.scala:57) finished in 0.270 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.807][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 2 finished: treeAggregate at StandardScaler.scala:57, took 0.289775 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.852][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Removed broadcast_2_piece0 on DESKTOP-ZY:49462 in memory (size: 4.2 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:58.999][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 92
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.000][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 97
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.037][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Removed broadcast_3_piece0 on DESKTOP-ZY:49462 in memory (size: 2014.0 B, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 91
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 86
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 69
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 93
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 83
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 117
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 105
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 104
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.095][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 84
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.095][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 76
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.095][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 80
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.095][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 114
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.095][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 68
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.095][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 107
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.095][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 73
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 98
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 100
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 112
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 70
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 75
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 77
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 109
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 71
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 94
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 116
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 67
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 85
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 113
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 96
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 103
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 99
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 106
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 111
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 115
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 102
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 78
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 108
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 118
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 87
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 110
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 119
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 90
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 101
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.099][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 81
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.099][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 88
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.099][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 82
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.099][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 89
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.113][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Removed broadcast_1_piece0 on DESKTOP-ZY:49462 in memory (size: 3.8 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.377][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: count at DNNClassification.scala:69
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.378][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Registering RDD 24 (count at DNNClassification.scala:69)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.379][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 3 (count at DNNClassification.scala:69) with 1 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.379][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 6 (count at DNNClassification.scala:69)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.379][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List(ShuffleMapStage 5)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.379][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List(ShuffleMapStage 5)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.380][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ShuffleMapStage 5 (MapPartitionsRDD[24] at count at DNNClassification.scala:69), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.383][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_5 stored as values in memory (estimated size 7.9 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.386][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_5_piece0 stored as bytes in memory (estimated size 4.2 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.386][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Added broadcast_5_piece0 in memory on DESKTOP-ZY:49462 (size: 4.2 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.388][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 5 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.390][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ShuffleMapStage 5 (MapPartitionsRDD[24] at count at DNNClassification.scala:69) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.390][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 5.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.391][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 0.0 in stage 5.0 (TID 17, localhost, executor driver, partition 0, PROCESS_LOCAL, 8629 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.392][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 1.0 in stage 5.0 (TID 18, localhost, executor driver, partition 1, PROCESS_LOCAL, 8646 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.392][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 2.0 in stage 5.0 (TID 19, localhost, executor driver, partition 2, PROCESS_LOCAL, 8629 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.393][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 3.0 in stage 5.0 (TID 20, localhost, executor driver, partition 3, PROCESS_LOCAL, 8646 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.394][INFO ][Executor task launch worker for task 17][Logging.scala:54]-Running task 0.0 in stage 5.0 (TID 17)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.394][INFO ][Executor task launch worker for task 18][Logging.scala:54]-Running task 1.0 in stage 5.0 (TID 18)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.396][INFO ][Executor task launch worker for task 19][Logging.scala:54]-Running task 2.0 in stage 5.0 (TID 19)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.396][INFO ][Executor task launch worker for task 20][Logging.scala:54]-Running task 3.0 in stage 5.0 (TID 20)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.450][INFO ][Executor task launch worker for task 17][Logging.scala:54]-Finished task 0.0 in stage 5.0 (TID 17). 1457 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.451][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 0.0 in stage 5.0 (TID 17) in 60 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.456][INFO ][Executor task launch worker for task 20][Logging.scala:54]-Finished task 3.0 in stage 5.0 (TID 20). 1457 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.459][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 3.0 in stage 5.0 (TID 20) in 65 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.466][INFO ][Executor task launch worker for task 18][Logging.scala:54]-Finished task 1.0 in stage 5.0 (TID 18). 1457 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.467][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 1.0 in stage 5.0 (TID 18) in 76 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.471][INFO ][Executor task launch worker for task 19][Logging.scala:54]-Finished task 2.0 in stage 5.0 (TID 19). 1414 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.472][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 2.0 in stage 5.0 (TID 19) in 80 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.472][INFO ][task-result-getter-0][Logging.scala:54]-Removed TaskSet 5.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.473][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ShuffleMapStage 5 (count at DNNClassification.scala:69) finished in 0.092 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.473][INFO ][dag-scheduler-event-loop][Logging.scala:54]-looking for newly runnable stages
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.473][INFO ][dag-scheduler-event-loop][Logging.scala:54]-running: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.473][INFO ][dag-scheduler-event-loop][Logging.scala:54]-waiting: Set(ResultStage 6)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.473][INFO ][dag-scheduler-event-loop][Logging.scala:54]-failed: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.474][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 6 (MapPartitionsRDD[27] at count at DNNClassification.scala:69), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.476][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_6 stored as values in memory (estimated size 7.1 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.478][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_6_piece0 stored as bytes in memory (estimated size 3.8 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.482][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Added broadcast_6_piece0 in memory on DESKTOP-ZY:49462 (size: 3.8 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.482][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 6 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.483][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[27] at count at DNNClassification.scala:69) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.483][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 6.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.485][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 0.0 in stage 6.0 (TID 21, localhost, executor driver, partition 0, ANY, 7767 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.486][INFO ][Executor task launch worker for task 21][Logging.scala:54]-Running task 0.0 in stage 6.0 (TID 21)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.502][INFO ][Executor task launch worker for task 21][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.503][INFO ][Executor task launch worker for task 21][Logging.scala:54]-Started 0 remote fetches in 1 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.508][INFO ][Executor task launch worker for task 21][Logging.scala:54]-Finished task 0.0 in stage 6.0 (TID 21). 1696 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.509][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 0.0 in stage 6.0 (TID 21) in 24 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.510][INFO ][task-result-getter-1][Logging.scala:54]-Removed TaskSet 6.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.510][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 6 (count at DNNClassification.scala:69) finished in 0.035 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.511][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 3 finished: count at DNNClassification.scala:69, took 0.133431 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.626][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 31.6235 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.713][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: count at DNNClassification.scala:80
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.714][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Registering RDD 33 (distinct at DNNClassification.scala:80)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.715][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 4 (count at DNNClassification.scala:80) with 4 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.715][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 8 (count at DNNClassification.scala:80)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.715][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List(ShuffleMapStage 7)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.715][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List(ShuffleMapStage 7)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.716][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ShuffleMapStage 7 (MapPartitionsRDD[33] at distinct at DNNClassification.scala:80), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.894][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_7 stored as values in memory (estimated size 7.3 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.927][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_7_piece0 stored as bytes in memory (estimated size 4.1 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.934][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Added broadcast_7_piece0 in memory on DESKTOP-ZY:49462 (size: 4.1 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.935][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 7 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.940][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ShuffleMapStage 7 (MapPartitionsRDD[33] at distinct at DNNClassification.scala:80) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.941][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 7.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.942][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 0.0 in stage 7.0 (TID 22, localhost, executor driver, partition 0, PROCESS_LOCAL, 9221 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.943][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 1.0 in stage 7.0 (TID 23, localhost, executor driver, partition 1, PROCESS_LOCAL, 9254 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.944][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 2.0 in stage 7.0 (TID 24, localhost, executor driver, partition 2, PROCESS_LOCAL, 9221 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.944][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 3.0 in stage 7.0 (TID 25, localhost, executor driver, partition 3, PROCESS_LOCAL, 9254 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.945][INFO ][Executor task launch worker for task 23][Logging.scala:54]-Running task 1.0 in stage 7.0 (TID 23)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.945][INFO ][Executor task launch worker for task 24][Logging.scala:54]-Running task 2.0 in stage 7.0 (TID 24)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.945][INFO ][Executor task launch worker for task 22][Logging.scala:54]-Running task 0.0 in stage 7.0 (TID 22)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.955][INFO ][Executor task launch worker for task 25][Logging.scala:54]-Running task 3.0 in stage 7.0 (TID 25)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:10:59.974][INFO ][Executor task launch worker for task 23][Logging.scala:54]-Code generated in 21.5682 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.240][INFO ][Executor task launch worker for task 25][Logging.scala:54]-Finished task 3.0 in stage 7.0 (TID 25). 1201 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.241][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 3.0 in stage 7.0 (TID 25) in 297 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.316][INFO ][Executor task launch worker for task 23][Logging.scala:54]-Finished task 1.0 in stage 7.0 (TID 23). 1244 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.321][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 1.0 in stage 7.0 (TID 23) in 378 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.413][INFO ][Executor task launch worker for task 22][Logging.scala:54]-Finished task 0.0 in stage 7.0 (TID 22). 1244 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.414][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 0.0 in stage 7.0 (TID 22) in 472 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.461][INFO ][Executor task launch worker for task 24][Logging.scala:54]-Finished task 2.0 in stage 7.0 (TID 24). 1330 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.463][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 2.0 in stage 7.0 (TID 24) in 520 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.464][INFO ][task-result-getter-1][Logging.scala:54]-Removed TaskSet 7.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.465][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ShuffleMapStage 7 (distinct at DNNClassification.scala:80) finished in 0.749 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.465][INFO ][dag-scheduler-event-loop][Logging.scala:54]-looking for newly runnable stages
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.465][INFO ][dag-scheduler-event-loop][Logging.scala:54]-running: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.466][INFO ][dag-scheduler-event-loop][Logging.scala:54]-waiting: Set(ResultStage 8)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.466][INFO ][dag-scheduler-event-loop][Logging.scala:54]-failed: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.468][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 8 (MapPartitionsRDD[35] at distinct at DNNClassification.scala:80), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.475][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_8 stored as values in memory (estimated size 3.9 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.477][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.3 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.478][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Added broadcast_8_piece0 in memory on DESKTOP-ZY:49462 (size: 2.3 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.478][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 8 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.480][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ResultStage 8 (MapPartitionsRDD[35] at distinct at DNNClassification.scala:80) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.480][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 8.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.481][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 1.0 in stage 8.0 (TID 26, localhost, executor driver, partition 1, PROCESS_LOCAL, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.482][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 2.0 in stage 8.0 (TID 27, localhost, executor driver, partition 2, PROCESS_LOCAL, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.647][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 3.0 in stage 8.0 (TID 28, localhost, executor driver, partition 3, PROCESS_LOCAL, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.647][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 0.0 in stage 8.0 (TID 29, localhost, executor driver, partition 0, ANY, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.648][INFO ][Executor task launch worker for task 28][Logging.scala:54]-Running task 3.0 in stage 8.0 (TID 28)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.648][INFO ][Executor task launch worker for task 29][Logging.scala:54]-Running task 0.0 in stage 8.0 (TID 29)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.648][INFO ][Executor task launch worker for task 26][Logging.scala:54]-Running task 1.0 in stage 8.0 (TID 26)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.648][INFO ][Executor task launch worker for task 27][Logging.scala:54]-Running task 2.0 in stage 8.0 (TID 27)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.650][INFO ][Executor task launch worker for task 29][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.650][INFO ][Executor task launch worker for task 28][Logging.scala:54]-Getting 0 non-empty blocks including 0 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.650][INFO ][Executor task launch worker for task 29][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.650][INFO ][Executor task launch worker for task 28][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.650][INFO ][Executor task launch worker for task 26][Logging.scala:54]-Getting 0 non-empty blocks including 0 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.650][INFO ][Executor task launch worker for task 26][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.650][INFO ][Executor task launch worker for task 27][Logging.scala:54]-Getting 0 non-empty blocks including 0 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.653][INFO ][Executor task launch worker for task 27][Logging.scala:54]-Started 0 remote fetches in 3 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.657][INFO ][Executor task launch worker for task 28][Logging.scala:54]-Finished task 3.0 in stage 8.0 (TID 28). 1133 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.658][INFO ][Executor task launch worker for task 29][Logging.scala:54]-Finished task 0.0 in stage 8.0 (TID 29). 1090 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.660][INFO ][Executor task launch worker for task 26][Logging.scala:54]-Finished task 1.0 in stage 8.0 (TID 26). 1090 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.661][INFO ][Executor task launch worker for task 27][Logging.scala:54]-Finished task 2.0 in stage 8.0 (TID 27). 1047 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.662][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 3.0 in stage 8.0 (TID 28) in 15 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.662][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 1.0 in stage 8.0 (TID 26) in 181 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.663][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 0.0 in stage 8.0 (TID 29) in 16 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.663][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 2.0 in stage 8.0 (TID 27) in 181 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.664][INFO ][task-result-getter-1][Logging.scala:54]-Removed TaskSet 8.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.664][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 8 (count at DNNClassification.scala:80) finished in 0.193 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.665][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 4 finished: count at DNNClassification.scala:80, took 0.951500 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.917][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 29.5682 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:00.955][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 24.3491 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.453][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: reduce at DNNClassification.scala:135
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.455][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Registering RDD 40 (map at DNNClassification.scala:95)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.456][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Registering RDD 42 (map at DNNClassification.scala:99)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.456][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 5 (reduce at DNNClassification.scala:135) with 4 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.456][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 11 (reduce at DNNClassification.scala:135)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.456][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List(ShuffleMapStage 10)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.456][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List(ShuffleMapStage 10)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.457][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ShuffleMapStage 9 (MapPartitionsRDD[40] at map at DNNClassification.scala:95), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.459][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_9 stored as values in memory (estimated size 9.8 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.462][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_9_piece0 stored as bytes in memory (estimated size 5.3 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.463][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Added broadcast_9_piece0 in memory on DESKTOP-ZY:49462 (size: 5.3 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.463][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 9 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.464][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ShuffleMapStage 9 (MapPartitionsRDD[40] at map at DNNClassification.scala:95) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.464][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 9.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.466][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 0.0 in stage 9.0 (TID 30, localhost, executor driver, partition 0, PROCESS_LOCAL, 12773 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.466][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 1.0 in stage 9.0 (TID 31, localhost, executor driver, partition 1, PROCESS_LOCAL, 12902 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.467][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 2.0 in stage 9.0 (TID 32, localhost, executor driver, partition 2, PROCESS_LOCAL, 12773 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.468][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 3.0 in stage 9.0 (TID 33, localhost, executor driver, partition 3, PROCESS_LOCAL, 12902 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.468][INFO ][Executor task launch worker for task 30][Logging.scala:54]-Running task 0.0 in stage 9.0 (TID 30)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.468][INFO ][Executor task launch worker for task 32][Logging.scala:54]-Running task 2.0 in stage 9.0 (TID 32)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.468][INFO ][Executor task launch worker for task 31][Logging.scala:54]-Running task 1.0 in stage 9.0 (TID 31)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.471][INFO ][Executor task launch worker for task 33][Logging.scala:54]-Running task 3.0 in stage 9.0 (TID 33)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.527][INFO ][Executor task launch worker for task 30][Logging.scala:54]-Finished task 0.0 in stage 9.0 (TID 30). 1244 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.528][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 0.0 in stage 9.0 (TID 30) in 62 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.546][INFO ][Executor task launch worker for task 32][Logging.scala:54]-Finished task 2.0 in stage 9.0 (TID 32). 1244 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.547][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 2.0 in stage 9.0 (TID 32) in 80 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.562][INFO ][Executor task launch worker for task 33][Logging.scala:54]-Finished task 3.0 in stage 9.0 (TID 33). 1244 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.563][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 3.0 in stage 9.0 (TID 33) in 96 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.576][INFO ][Executor task launch worker for task 31][Logging.scala:54]-Finished task 1.0 in stage 9.0 (TID 31). 1244 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.577][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 1.0 in stage 9.0 (TID 31) in 111 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.577][INFO ][task-result-getter-1][Logging.scala:54]-Removed TaskSet 9.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.578][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ShuffleMapStage 9 (map at DNNClassification.scala:95) finished in 0.120 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.578][INFO ][dag-scheduler-event-loop][Logging.scala:54]-looking for newly runnable stages
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.578][INFO ][dag-scheduler-event-loop][Logging.scala:54]-running: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.578][INFO ][dag-scheduler-event-loop][Logging.scala:54]-waiting: Set(ShuffleMapStage 10, ResultStage 11)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.578][INFO ][dag-scheduler-event-loop][Logging.scala:54]-failed: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.579][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ShuffleMapStage 10 (MapPartitionsRDD[42] at map at DNNClassification.scala:99), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.591][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_10 stored as values in memory (estimated size 14.8 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.593][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_10_piece0 stored as bytes in memory (estimated size 7.5 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.597][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Added broadcast_10_piece0 in memory on DESKTOP-ZY:49462 (size: 7.5 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.598][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 10 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.598][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ShuffleMapStage 10 (MapPartitionsRDD[42] at map at DNNClassification.scala:99) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.598][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 10.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.600][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 0.0 in stage 10.0 (TID 34, localhost, executor driver, partition 0, PROCESS_LOCAL, 7651 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.600][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 2.0 in stage 10.0 (TID 35, localhost, executor driver, partition 2, PROCESS_LOCAL, 7651 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.601][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 3.0 in stage 10.0 (TID 36, localhost, executor driver, partition 3, PROCESS_LOCAL, 7651 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.601][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 1.0 in stage 10.0 (TID 37, localhost, executor driver, partition 1, ANY, 7651 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.602][INFO ][Executor task launch worker for task 34][Logging.scala:54]-Running task 0.0 in stage 10.0 (TID 34)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.602][INFO ][Executor task launch worker for task 37][Logging.scala:54]-Running task 1.0 in stage 10.0 (TID 37)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.602][INFO ][Executor task launch worker for task 35][Logging.scala:54]-Running task 2.0 in stage 10.0 (TID 35)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.608][INFO ][Executor task launch worker for task 34][Logging.scala:54]-Getting 0 non-empty blocks including 0 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.608][INFO ][Executor task launch worker for task 34][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.609][INFO ][Executor task launch worker for task 37][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.609][INFO ][Executor task launch worker for task 37][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.612][INFO ][Executor task launch worker for task 35][Logging.scala:54]-Getting 0 non-empty blocks including 0 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.602][INFO ][Executor task launch worker for task 36][Logging.scala:54]-Running task 3.0 in stage 10.0 (TID 36)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.613][INFO ][Executor task launch worker for task 35][Logging.scala:54]-Started 0 remote fetches in 1 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.623][INFO ][Executor task launch worker for task 36][Logging.scala:54]-Getting 0 non-empty blocks including 0 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.624][INFO ][Executor task launch worker for task 36][Logging.scala:54]-Started 0 remote fetches in 1 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.655][INFO ][Executor task launch worker for task 34][Logging.scala:54]-Finished task 0.0 in stage 10.0 (TID 34). 1459 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.656][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 0.0 in stage 10.0 (TID 34) in 57 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.659][INFO ][Executor task launch worker for task 36][Logging.scala:54]-Finished task 3.0 in stage 10.0 (TID 36). 1416 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.660][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 3.0 in stage 10.0 (TID 36) in 59 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.664][INFO ][Executor task launch worker for task 35][Logging.scala:54]-Finished task 2.0 in stage 10.0 (TID 35). 1459 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.665][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 2.0 in stage 10.0 (TID 35) in 65 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:01.680][INFO ][Executor task launch worker for task 37][Nd4jBackend.java:194]-Loaded [CpuBackend] backend
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:02.276][INFO ][Executor task launch worker for task 37][NativeOpsHolder.java:49]-Number of threads used for NativeOps: 2
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:02.767][INFO ][Executor task launch worker for task 37][Reflections.java:229]-Reflections took 423 ms to scan 1 urls, producing 29 keys and 189 values
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.506][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 208
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.506][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 176
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.506][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 126
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.507][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 156
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.507][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 241
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.507][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 250
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.507][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 194
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.507][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 223
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.507][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 231
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.507][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 155
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.507][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 203
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.508][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 218
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.508][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 164
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.508][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 195
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.508][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 251
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.508][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 154
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.508][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 222
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.508][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 247
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.660][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Removed broadcast_6_piece0 on DESKTOP-ZY:49462 in memory (size: 3.8 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.679][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 192
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.679][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 233
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.680][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 169
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.680][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 128
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.680][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 198
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.680][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 133
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.680][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 171
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.806][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Removed broadcast_5_piece0 on DESKTOP-ZY:49462 in memory (size: 4.2 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.826][INFO ][Executor task launch worker for task 37][Nd4jBlas.java:36]-Number of threads used for BLAS: 2
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.831][INFO ][Executor task launch worker for task 37][DefaultOpExecutioner.java:544]-Backend used: [CPU]; OS: [Windows 10]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.832][INFO ][Executor task launch worker for task 37][DefaultOpExecutioner.java:545]-Cores: [4]; Memory: [3.5GB];
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.832][INFO ][Executor task launch worker for task 37][DefaultOpExecutioner.java:546]-Blas vendor: [OPENBLAS]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.927][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 214
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.927][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 225
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.928][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 226
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.928][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 254
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.928][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 166
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:03.976][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Removed broadcast_4_piece0 on DESKTOP-ZY:49462 in memory (size: 4.6 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.050][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 189
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.051][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 121
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.051][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 204
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.051][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 230
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.051][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 187
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.051][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 138
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.052][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 246
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.052][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 197
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.052][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 202
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.052][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 206
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.052][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 248
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.052][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 242
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.052][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 148
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.052][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 219
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.052][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 249
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.052][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 205
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 124
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 215
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 238
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 196
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 143
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 140
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 255
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 260
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 136
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 132
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 252
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 217
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.053][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 127
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.054][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 142
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.054][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 144
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.054][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 180
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.054][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 158
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.054][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 191
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.055][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 163
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.055][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 182
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.055][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 259
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.055][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 239
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.055][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 175
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.055][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 228
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.055][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 224
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.056][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 240
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.056][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 129
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.056][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.056][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 139
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.056][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 172
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.056][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 157
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.056][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 190
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.057][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 199
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.057][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 232
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.057][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 152
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.057][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 151
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.057][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 146
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.058][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 236
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.058][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 207
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.059][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 210
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.059][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 130
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.059][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 183
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.059][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 165
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.059][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 234
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.059][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 184
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.059][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 244
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.060][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 149
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.060][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 147
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.060][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 221
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.060][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 201
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.062][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned shuffle 2
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.062][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 263
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.062][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 257
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.062][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 174
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.062][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 162
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.063][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 200
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.063][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 160
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.063][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 145
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.063][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 212
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.063][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 159
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.063][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 141
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.063][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 193
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.063][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 186
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.063][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 134
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.080][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Removed broadcast_8_piece0 on DESKTOP-ZY:49462 in memory (size: 2.3 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.122][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 227
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.123][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 170
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.123][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 123
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.123][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 243
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.123][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 256
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.123][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 181
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.123][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 131
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.123][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 153
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.124][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 173
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.124][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 209
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.124][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 211
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.124][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 122
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.124][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 125
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.124][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 188
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.124][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 216
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.124][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 258
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.125][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 229
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.125][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 262
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.125][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 235
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.161][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Removed broadcast_9_piece0 on DESKTOP-ZY:49462 in memory (size: 5.3 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.228][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 213
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.229][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 177
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.229][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 120
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.262][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Removed broadcast_7_piece0 on DESKTOP-ZY:49462 in memory (size: 4.1 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.281][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 150
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.282][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 185
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.284][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned shuffle 3
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.285][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 261
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.286][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 135
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.287][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 168
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.288][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 220
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.288][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 167
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.288][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 178
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.288][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 253
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.288][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 237
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.289][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 179
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.289][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 137
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.289][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 245
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:04.973][INFO ][Executor task launch worker for task 37][Reflections.java:229]-Reflections took 7 ms to scan 2 urls, producing 11 keys and 15 values
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:05.837][INFO ][Executor task launch worker for task 37][Reflections.java:229]-Reflections took 498 ms to scan 1 urls, producing 373 keys and 1449 values
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.324][INFO ][Executor task launch worker for task 37][Logging.scala:54]-Finished task 1.0 in stage 10.0 (TID 37). 1588 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.325][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 1.0 in stage 10.0 (TID 37) in 28724 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.325][INFO ][task-result-getter-1][Logging.scala:54]-Removed TaskSet 10.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.325][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ShuffleMapStage 10 (map at DNNClassification.scala:99) finished in 28.745 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.325][INFO ][dag-scheduler-event-loop][Logging.scala:54]-looking for newly runnable stages
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.327][INFO ][dag-scheduler-event-loop][Logging.scala:54]-running: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.327][INFO ][dag-scheduler-event-loop][Logging.scala:54]-waiting: Set(ResultStage 11)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.327][INFO ][dag-scheduler-event-loop][Logging.scala:54]-failed: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.327][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 11 (MapPartitionsRDD[44] at map at DNNClassification.scala:133), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.330][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_11 stored as values in memory (estimated size 7.8 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.331][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_11_piece0 stored as bytes in memory (estimated size 4.1 KB, free 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.332][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Added broadcast_11_piece0 in memory on DESKTOP-ZY:49462 (size: 4.1 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.333][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 11 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.333][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ResultStage 11 (MapPartitionsRDD[44] at map at DNNClassification.scala:133) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.333][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 11.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.334][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 0.0 in stage 11.0 (TID 38, localhost, executor driver, partition 0, PROCESS_LOCAL, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.335][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 2.0 in stage 11.0 (TID 39, localhost, executor driver, partition 2, PROCESS_LOCAL, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.335][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 3.0 in stage 11.0 (TID 40, localhost, executor driver, partition 3, PROCESS_LOCAL, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.336][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 1.0 in stage 11.0 (TID 41, localhost, executor driver, partition 1, ANY, 7662 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.336][INFO ][Executor task launch worker for task 38][Logging.scala:54]-Running task 0.0 in stage 11.0 (TID 38)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.336][INFO ][Executor task launch worker for task 40][Logging.scala:54]-Running task 3.0 in stage 11.0 (TID 40)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.337][INFO ][Executor task launch worker for task 41][Logging.scala:54]-Running task 1.0 in stage 11.0 (TID 41)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.336][INFO ][Executor task launch worker for task 39][Logging.scala:54]-Running task 2.0 in stage 11.0 (TID 39)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.339][INFO ][Executor task launch worker for task 38][Logging.scala:54]-Getting 0 non-empty blocks including 0 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.339][INFO ][Executor task launch worker for task 38][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.340][INFO ][Executor task launch worker for task 39][Logging.scala:54]-Getting 0 non-empty blocks including 0 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.341][INFO ][Executor task launch worker for task 39][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.341][INFO ][Executor task launch worker for task 38][Logging.scala:54]-Finished task 0.0 in stage 11.0 (TID 38). 1110 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.341][INFO ][Executor task launch worker for task 41][Logging.scala:54]-Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.341][INFO ][Executor task launch worker for task 40][Logging.scala:54]-Getting 0 non-empty blocks including 0 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.341][INFO ][Executor task launch worker for task 41][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.341][INFO ][Executor task launch worker for task 39][Logging.scala:54]-Finished task 2.0 in stage 11.0 (TID 39). 1110 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.342][INFO ][Executor task launch worker for task 40][Logging.scala:54]-Started 0 remote fetches in 2 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.342][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 0.0 in stage 11.0 (TID 38) in 8 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.342][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 2.0 in stage 11.0 (TID 39) in 7 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.343][INFO ][Executor task launch worker for task 40][Logging.scala:54]-Finished task 3.0 in stage 11.0 (TID 40). 1110 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.344][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 3.0 in stage 11.0 (TID 40) in 9 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.466][INFO ][Executor task launch worker for task 41][Logging.scala:54]-Finished task 1.0 in stage 11.0 (TID 41). 11566 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.468][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 1.0 in stage 11.0 (TID 41) in 132 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.468][INFO ][task-result-getter-1][Logging.scala:54]-Removed TaskSet 11.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.469][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 11 (reduce at DNNClassification.scala:135) finished in 0.140 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.469][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 5 finished: reduce at DNNClassification.scala:135, took 29.015083 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:30.920][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 24.1788 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.191][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 32.5269 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.220][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 23.0427 ms
featureoutput
standardOutput
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.400][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 21.572 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.431][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 18.3576 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.447][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: show at DnnClassificationBehavior.scala:145
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.448][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Registering RDD 54 (rdd at ClassificationModelApply.scala:76)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.449][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 6 (show at DnnClassificationBehavior.scala:145) with 1 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.450][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 13 (show at DnnClassificationBehavior.scala:145)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.450][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List(ShuffleMapStage 12)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.451][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List(ShuffleMapStage 12)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.452][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ShuffleMapStage 12 (MapPartitionsRDD[54] at rdd at ClassificationModelApply.scala:76), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.469][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_12 stored as values in memory (estimated size 38.5 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.472][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_12_piece0 stored as bytes in memory (estimated size 17.2 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.473][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Added broadcast_12_piece0 in memory on DESKTOP-ZY:49462 (size: 17.2 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.474][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 12 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.474][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ShuffleMapStage 12 (MapPartitionsRDD[54] at rdd at ClassificationModelApply.scala:76) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.474][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 12.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.475][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 0.0 in stage 12.0 (TID 42, localhost, executor driver, partition 0, PROCESS_LOCAL, 18212 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.476][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 1.0 in stage 12.0 (TID 43, localhost, executor driver, partition 1, PROCESS_LOCAL, 18488 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.476][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 2.0 in stage 12.0 (TID 44, localhost, executor driver, partition 2, PROCESS_LOCAL, 18212 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.477][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 3.0 in stage 12.0 (TID 45, localhost, executor driver, partition 3, PROCESS_LOCAL, 18488 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.477][INFO ][Executor task launch worker for task 42][Logging.scala:54]-Running task 0.0 in stage 12.0 (TID 42)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.477][INFO ][Executor task launch worker for task 44][Logging.scala:54]-Running task 2.0 in stage 12.0 (TID 44)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.477][INFO ][Executor task launch worker for task 45][Logging.scala:54]-Running task 3.0 in stage 12.0 (TID 45)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.477][INFO ][Executor task launch worker for task 43][Logging.scala:54]-Running task 1.0 in stage 12.0 (TID 43)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.521][INFO ][Executor task launch worker for task 42][Logging.scala:54]-Code generated in 24.337 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:31.577][INFO ][Executor task launch worker for task 44][Logging.scala:54]-Code generated in 32.0981 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.818][INFO ][Executor task launch worker for task 42][Logging.scala:54]-Code generated in 71.1471 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.891][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 311
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.891][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 279
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.891][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 304
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.891][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 321
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.891][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 290
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.891][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 296
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.891][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 314
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.891][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 276
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.894][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Removed broadcast_10_piece0 on DESKTOP-ZY:49462 in memory (size: 7.5 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.897][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 277
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.897][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 275
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.897][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 326
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.897][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 340
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.897][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 297
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 267
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 269
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 294
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 293
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 327
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 288
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 300
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 280
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 271
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.899][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 265
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.899][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 274
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.899][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 281
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.899][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 273
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.899][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 287
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.899][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 342
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.899][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 328
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 310
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 289
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 264
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 324
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 292
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 319
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 316
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 302
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 336
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 317
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 322
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 312
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 306
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 313
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 282
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 343
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 283
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 285
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 331
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 320
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 308
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 270
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 307
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 268
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 303
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 338
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 284
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 335
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 315
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 301
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 278
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.904][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 298
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.906][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Removed broadcast_11_piece0 on DESKTOP-ZY:49462 in memory (size: 4.1 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.908][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 325
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.909][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned shuffle 4
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.909][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 332
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.909][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 330
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.909][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 272
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.909][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 339
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.910][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned shuffle 5
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.910][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 305
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.910][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 309
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.910][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 334
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.910][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 266
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.910][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 299
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.910][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 329
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.911][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 333
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.911][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 318
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.911][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 286
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.911][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 295
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.911][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 291
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.911][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 337
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:32.911][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 323
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.096][INFO ][Executor task launch worker for task 42][Logging.scala:54]-Finished task 0.0 in stage 12.0 (TID 42). 1450 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.097][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 0.0 in stage 12.0 (TID 42) in 1622 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.108][INFO ][Executor task launch worker for task 45][Logging.scala:54]-Finished task 3.0 in stage 12.0 (TID 45). 1450 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.108][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 3.0 in stage 12.0 (TID 45) in 1632 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.123][INFO ][Executor task launch worker for task 44][Logging.scala:54]-Finished task 2.0 in stage 12.0 (TID 44). 1407 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.124][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 2.0 in stage 12.0 (TID 44) in 1648 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.272][INFO ][Executor task launch worker for task 43][Logging.scala:54]-Finished task 1.0 in stage 12.0 (TID 43). 1407 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.272][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 1.0 in stage 12.0 (TID 43) in 1797 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.273][INFO ][task-result-getter-1][Logging.scala:54]-Removed TaskSet 12.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.273][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ShuffleMapStage 12 (rdd at ClassificationModelApply.scala:76) finished in 1.820 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.273][INFO ][dag-scheduler-event-loop][Logging.scala:54]-looking for newly runnable stages
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.273][INFO ][dag-scheduler-event-loop][Logging.scala:54]-running: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.273][INFO ][dag-scheduler-event-loop][Logging.scala:54]-waiting: Set(ResultStage 13)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.274][INFO ][dag-scheduler-event-loop][Logging.scala:54]-failed: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.274][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 13 (MapPartitionsRDD[64] at show at DnnClassificationBehavior.scala:145), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.304][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_13 stored as values in memory (estimated size 53.5 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.306][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_13_piece0 stored as bytes in memory (estimated size 22.4 KB, free 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.314][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Added broadcast_13_piece0 in memory on DESKTOP-ZY:49462 (size: 22.4 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.315][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 13 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.319][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ResultStage 13 (MapPartitionsRDD[64] at show at DnnClassificationBehavior.scala:145) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.319][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 13.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.320][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 0.0 in stage 13.0 (TID 46, localhost, executor driver, partition 0, ANY, 7767 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.321][INFO ][Executor task launch worker for task 46][Logging.scala:54]-Running task 0.0 in stage 13.0 (TID 46)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.330][INFO ][Executor task launch worker for task 46][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.330][INFO ][Executor task launch worker for task 46][Logging.scala:54]-Started 0 remote fetches in 1 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.360][INFO ][Executor task launch worker for task 46][Logging.scala:54]-Code generated in 15.2884 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.392][INFO ][Executor task launch worker for task 46][Logging.scala:54]-Code generated in 22.7106 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.486][INFO ][Executor task launch worker for task 46][Logging.scala:54]-Code generated in 56.874 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.492][INFO ][Executor task launch worker for task 46][Logging.scala:54]-Finished task 0.0 in stage 13.0 (TID 46). 1964 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.493][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 0.0 in stage 13.0 (TID 46) in 173 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.493][INFO ][task-result-getter-3][Logging.scala:54]-Removed TaskSet 13.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.493][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 13 (show at DnnClassificationBehavior.scala:145) finished in 0.218 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.494][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 6 finished: show at DnnClassificationBehavior.scala:145, took 2.045820 s
+---+---+------+----------+---------+-----------+--------------------+--------+--------+--------------------+
|150| 4|setosa|versicolor|virginica|predicition| probability|prob_ww0|prob_ww1| prob_ww2|
+---+---+------+----------+---------+-----------+--------------------+--------+--------+--------------------+
|5.7|3.8| 1.7| 0.3| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|5.5|3.5| 1.3| 0.2| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|5.0|3.6| 1.4| 0.2| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|5.0|3.4| 1.6| 0.4| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|5.8|4.0| 1.2| 0.2| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|5.2|4.1| 1.5| 0.1| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|4.7|3.2| 1.6| 0.2| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|4.8|3.1| 1.6| 0.2| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|5.1|3.5| 1.4| 0.3| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|6.5|2.8| 4.6| 1.5| ww1| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|5.6|3.0| 4.5| 1.5| ww1| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|5.1|3.8| 1.6| 0.2| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|5.9|3.2| 4.8| 1.8| ww1| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|6.2|2.2| 4.5| 1.5| ww1| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|4.4|3.0| 1.3| 0.2| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|5.0|2.0| 3.5| 1.0| ww1| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|4.6|3.2| 1.4| 0.2| ww0| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|5.2|2.7| 3.9| 1.4| ww1| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|6.4|3.2| 4.5| 1.5| ww1| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
|6.0|3.4| 4.5| 1.6| ww1| ww0|[0.9995,0.972,0.0...| 0.9995| 0.972|0.028000000000000025|
+---+---+------+----------+---------+-----------+--------------------+--------+--------+--------------------+
only showing top 20 rows
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.546][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 10.9882 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.572][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: collect at DataToJson.scala:75
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.573][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 7 (collect at DataToJson.scala:75) with 4 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.577][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 15 (collect at DataToJson.scala:75)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.577][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List(ShuffleMapStage 14)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.577][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.578][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 15 (MapPartitionsRDD[66] at collect at DataToJson.scala:75), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.584][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_14 stored as values in memory (estimated size 47.7 KB, free 1985.2 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.586][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_14_piece0 stored as bytes in memory (estimated size 20.7 KB, free 1985.2 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.587][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Added broadcast_14_piece0 in memory on DESKTOP-ZY:49462 (size: 20.7 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.589][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 14 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.593][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ResultStage 15 (MapPartitionsRDD[66] at collect at DataToJson.scala:75) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.593][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 15.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.594][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 0.0 in stage 15.0 (TID 47, localhost, executor driver, partition 0, ANY, 7767 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.594][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 1.0 in stage 15.0 (TID 48, localhost, executor driver, partition 1, ANY, 7767 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.595][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 2.0 in stage 15.0 (TID 49, localhost, executor driver, partition 2, ANY, 7767 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.595][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 3.0 in stage 15.0 (TID 50, localhost, executor driver, partition 3, ANY, 7767 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.595][INFO ][Executor task launch worker for task 47][Logging.scala:54]-Running task 0.0 in stage 15.0 (TID 47)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.595][INFO ][Executor task launch worker for task 49][Logging.scala:54]-Running task 2.0 in stage 15.0 (TID 49)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.595][INFO ][Executor task launch worker for task 48][Logging.scala:54]-Running task 1.0 in stage 15.0 (TID 48)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.597][INFO ][Executor task launch worker for task 50][Logging.scala:54]-Running task 3.0 in stage 15.0 (TID 50)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.603][INFO ][Executor task launch worker for task 50][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.603][INFO ][Executor task launch worker for task 50][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.612][INFO ][Executor task launch worker for task 47][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.613][INFO ][Executor task launch worker for task 47][Logging.scala:54]-Started 0 remote fetches in 1 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.617][INFO ][Executor task launch worker for task 49][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.617][INFO ][Executor task launch worker for task 49][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.638][INFO ][Executor task launch worker for task 48][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.638][INFO ][Executor task launch worker for task 48][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.695][INFO ][Executor task launch worker for task 50][Logging.scala:54]-Finished task 3.0 in stage 15.0 (TID 50). 2649 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.695][INFO ][Executor task launch worker for task 49][Logging.scala:54]-Finished task 2.0 in stage 15.0 (TID 49). 2615 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.695][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 3.0 in stage 15.0 (TID 50) in 100 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.696][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 2.0 in stage 15.0 (TID 49) in 101 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.696][INFO ][Executor task launch worker for task 47][Logging.scala:54]-Finished task 0.0 in stage 15.0 (TID 47). 2634 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.697][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 0.0 in stage 15.0 (TID 47) in 103 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.711][INFO ][Executor task launch worker for task 48][Logging.scala:54]-Finished task 1.0 in stage 15.0 (TID 48). 2641 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.712][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 1.0 in stage 15.0 (TID 48) in 118 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.712][INFO ][task-result-getter-3][Logging.scala:54]-Removed TaskSet 15.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.712][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 15 (collect at DataToJson.scala:75) finished in 0.132 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.713][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 7 finished: collect at DataToJson.scala:75, took 0.139890 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.790][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 14.8067 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.814][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: count at DataToJson.scala:168
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.816][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Registering RDD 69 (count at DataToJson.scala:168)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.817][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 8 (count at DataToJson.scala:168) with 1 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.817][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 18 (count at DataToJson.scala:168)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.817][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List(ShuffleMapStage 17)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.818][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List(ShuffleMapStage 17)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.818][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ShuffleMapStage 17 (MapPartitionsRDD[69] at count at DataToJson.scala:168), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.824][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_15 stored as values in memory (estimated size 50.2 KB, free 1985.1 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.826][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_15_piece0 stored as bytes in memory (estimated size 22.0 KB, free 1985.1 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.827][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Added broadcast_15_piece0 in memory on DESKTOP-ZY:49462 (size: 22.0 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.827][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 15 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.829][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 4 missing tasks from ShuffleMapStage 17 (MapPartitionsRDD[69] at count at DataToJson.scala:168) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.830][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 17.0 with 4 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.830][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 0.0 in stage 17.0 (TID 51, localhost, executor driver, partition 0, ANY, 7756 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.831][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 1.0 in stage 17.0 (TID 52, localhost, executor driver, partition 1, ANY, 7756 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.831][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 2.0 in stage 17.0 (TID 53, localhost, executor driver, partition 2, ANY, 7756 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.831][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 3.0 in stage 17.0 (TID 54, localhost, executor driver, partition 3, ANY, 7756 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.832][INFO ][Executor task launch worker for task 51][Logging.scala:54]-Running task 0.0 in stage 17.0 (TID 51)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.833][INFO ][Executor task launch worker for task 52][Logging.scala:54]-Running task 1.0 in stage 17.0 (TID 52)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.834][INFO ][Executor task launch worker for task 53][Logging.scala:54]-Running task 2.0 in stage 17.0 (TID 53)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.834][INFO ][Executor task launch worker for task 54][Logging.scala:54]-Running task 3.0 in stage 17.0 (TID 54)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.843][INFO ][Executor task launch worker for task 54][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.843][INFO ][Executor task launch worker for task 54][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.846][INFO ][Executor task launch worker for task 51][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.846][INFO ][Executor task launch worker for task 51][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.847][INFO ][Executor task launch worker for task 52][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.847][INFO ][Executor task launch worker for task 52][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.856][INFO ][Executor task launch worker for task 53][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.856][INFO ][Executor task launch worker for task 53][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.971][INFO ][Executor task launch worker for task 52][Logging.scala:54]-Finished task 1.0 in stage 17.0 (TID 52). 2196 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.972][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 1.0 in stage 17.0 (TID 52) in 140 ms on localhost (executor driver) (1/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.975][INFO ][Executor task launch worker for task 53][Logging.scala:54]-Finished task 2.0 in stage 17.0 (TID 53). 2153 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.975][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 2.0 in stage 17.0 (TID 53) in 144 ms on localhost (executor driver) (2/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.980][INFO ][Executor task launch worker for task 51][Logging.scala:54]-Finished task 0.0 in stage 17.0 (TID 51). 2153 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:33.980][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 0.0 in stage 17.0 (TID 51) in 150 ms on localhost (executor driver) (3/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.000][INFO ][Executor task launch worker for task 54][Logging.scala:54]-Finished task 3.0 in stage 17.0 (TID 54). 2196 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.001][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 3.0 in stage 17.0 (TID 54) in 170 ms on localhost (executor driver) (4/4)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.001][INFO ][task-result-getter-3][Logging.scala:54]-Removed TaskSet 17.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.001][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ShuffleMapStage 17 (count at DataToJson.scala:168) finished in 0.181 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.002][INFO ][dag-scheduler-event-loop][Logging.scala:54]-looking for newly runnable stages
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.002][INFO ][dag-scheduler-event-loop][Logging.scala:54]-running: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.002][INFO ][dag-scheduler-event-loop][Logging.scala:54]-waiting: Set(ResultStage 18)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.002][INFO ][dag-scheduler-event-loop][Logging.scala:54]-failed: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.002][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 18 (MapPartitionsRDD[72] at count at DataToJson.scala:168), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.004][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_16 stored as values in memory (estimated size 7.1 KB, free 1985.1 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.006][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_16_piece0 stored as bytes in memory (estimated size 3.8 KB, free 1985.1 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.007][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Added broadcast_16_piece0 in memory on DESKTOP-ZY:49462 (size: 3.8 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.007][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 16 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.008][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ResultStage 18 (MapPartitionsRDD[72] at count at DataToJson.scala:168) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.008][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 18.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.009][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 0.0 in stage 18.0 (TID 55, localhost, executor driver, partition 0, ANY, 7767 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.009][INFO ][Executor task launch worker for task 55][Logging.scala:54]-Running task 0.0 in stage 18.0 (TID 55)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.012][INFO ][Executor task launch worker for task 55][Logging.scala:54]-Getting 4 non-empty blocks including 4 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.012][INFO ][Executor task launch worker for task 55][Logging.scala:54]-Started 0 remote fetches in 0 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.018][INFO ][Executor task launch worker for task 55][Logging.scala:54]-Finished task 0.0 in stage 18.0 (TID 55). 1739 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.019][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 0.0 in stage 18.0 (TID 55) in 10 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.019][INFO ][task-result-getter-0][Logging.scala:54]-Removed TaskSet 18.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.019][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 18 (count at DataToJson.scala:168) finished in 0.016 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:34.020][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 8 finished: count at DataToJson.scala:168, took 0.205364 s
log4j:WARN No appenders could be found for logger (org.apache.htrace.core.Tracer).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.736][INFO ][http-nio-20779-exec-1][Configuration.java:1274]-mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.744][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.747][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.747][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.828][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: runJob at SparkHadoopWriter.scala:78
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.829][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 9 (runJob at SparkHadoopWriter.scala:78) with 1 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.829][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 19 (runJob at SparkHadoopWriter.scala:78)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.829][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.829][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.829][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 19 (MapPartitionsRDD[74] at saveAsTextFile at ReadWrite.scala:441), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.844][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_17 stored as values in memory (estimated size 75.6 KB, free 1985.0 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.846][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_17_piece0 stored as bytes in memory (estimated size 27.2 KB, free 1985.0 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.846][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Added broadcast_17_piece0 in memory on DESKTOP-ZY:49462 (size: 27.2 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.847][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 17 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.848][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ResultStage 19 (MapPartitionsRDD[74] at saveAsTextFile at ReadWrite.scala:441) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.848][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 19.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.849][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 0.0 in stage 19.0 (TID 56, localhost, executor driver, partition 0, PROCESS_LOCAL, 8161 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.849][INFO ][Executor task launch worker for task 56][Logging.scala:54]-Running task 0.0 in stage 19.0 (TID 56)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.925][INFO ][Executor task launch worker for task 56][Logging.scala:54]-Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.926][INFO ][Executor task launch worker for task 56][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:35.926][INFO ][Executor task launch worker for task 56][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.069][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 373
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.069][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 382
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.069][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 415
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.069][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 421
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.069][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 352
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.069][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 354
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.069][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 413
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.069][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 420
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.069][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 462
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.070][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 448
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.070][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 408
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.070][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 379
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.071][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Removed broadcast_16_piece0 on DESKTOP-ZY:49462 in memory (size: 3.8 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.073][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 370
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.073][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 367
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.075][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 447
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.075][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 473
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.076][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 450
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.076][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 465
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.076][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 369
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.076][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 482
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.076][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 458
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.076][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 407
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.076][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 394
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.076][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 349
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.076][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 435
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.076][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 426
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 463
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 405
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 348
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 359
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 484
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 375
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 439
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 406
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 377
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 442
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 457
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.077][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 396
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.078][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 452
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.078][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 356
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.078][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 357
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.078][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 410
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.078][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 478
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.078][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 404
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.078][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 389
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.079][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 391
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.079][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 358
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.079][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 431
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.079][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 414
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.079][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 470
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.079][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 365
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.079][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 477
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.079][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 485
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.079][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 393
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.079][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 430
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.079][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 427
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.080][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 351
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.080][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 399
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.080][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 480
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.087][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Removed broadcast_15_piece0 on DESKTOP-ZY:49462 in memory (size: 22.0 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.093][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 385
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.093][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 429
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.093][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 475
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 469
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 353
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 371
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 366
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 474
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 451
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 466
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.094][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 419
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.095][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 467
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.095][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 350
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.095][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 449
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.095][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 440
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 436
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 372
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 446
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 432
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 443
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 398
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 489
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 417
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.096][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 390
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned shuffle 7
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.097][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 437
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 441
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 460
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 459
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 363
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.098][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 378
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.099][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 471
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.099][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 425
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.099][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 380
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.103][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Removed broadcast_12_piece0 on DESKTOP-ZY:49462 in memory (size: 17.2 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.108][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 411
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.108][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 384
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.109][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 445
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.109][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 360
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.109][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 368
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.109][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 374
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.109][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 395
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.109][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 376
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.109][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 364
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.109][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 383
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.109][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 397
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.110][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 456
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.110][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 412
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.110][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 428
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.110][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 401
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.110][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 402
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.110][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 416
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.110][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 488
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.110][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 476
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.110][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 490
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.110][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 381
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 423
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 444
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 361
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 455
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 409
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 355
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 472
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 422
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 434
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 481
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 424
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 433
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.111][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 486
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.112][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 386
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.112][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 479
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.116][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Removed broadcast_13_piece0 on DESKTOP-ZY:49462 in memory (size: 22.4 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.119][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 403
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.120][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 453
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.120][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 454
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.120][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 483
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.120][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 392
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.120][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 487
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.120][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 362
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.120][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 418
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.122][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Removed broadcast_14_piece0 on DESKTOP-ZY:49462 in memory (size: 20.7 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.124][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 438
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.125][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 464
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.125][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 468
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.125][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 461
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.125][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 388
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.125][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 387
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.451][INFO ][Executor task launch worker for task 56][FileOutputCommitter.java:583]-Saved output of task 'attempt_20200102121135_0074_m_000000_0' to hdfs://192.168.98.44:8020/nari/bdp/mine/train/model/837030eeb4ea4a389366202d7daabcf6/metadata/_temporary/0/task_20200102121135_0074_m_000000
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.452][INFO ][Executor task launch worker for task 56][Logging.scala:54]-attempt_20200102121135_0074_m_000000_0: Committed
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.454][INFO ][Executor task launch worker for task 56][Logging.scala:54]-Finished task 0.0 in stage 19.0 (TID 56). 1115 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.455][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 0.0 in stage 19.0 (TID 56) in 606 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.455][INFO ][task-result-getter-2][Logging.scala:54]-Removed TaskSet 19.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.456][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 19 (runJob at SparkHadoopWriter.scala:78) finished in 0.626 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.456][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 9 finished: runJob at SparkHadoopWriter.scala:78, took 0.628634 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.490][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job job_20200102121135_0074 committed.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.510][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.511][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.511][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.525][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: runJob at SparkHadoopWriter.scala:78
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.525][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 10 (runJob at SparkHadoopWriter.scala:78) with 1 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.525][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 20 (runJob at SparkHadoopWriter.scala:78)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.525][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.526][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.526][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 20 (MapPartitionsRDD[76] at saveAsTextFile at ReadWrite.scala:441), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.537][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_18 stored as values in memory (estimated size 75.6 KB, free 1985.2 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.538][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_18_piece0 stored as bytes in memory (estimated size 27.3 KB, free 1985.2 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.539][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Added broadcast_18_piece0 in memory on DESKTOP-ZY:49462 (size: 27.3 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.540][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 18 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.540][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ResultStage 20 (MapPartitionsRDD[76] at saveAsTextFile at ReadWrite.scala:441) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.540][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 20.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.541][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 0.0 in stage 20.0 (TID 57, localhost, executor driver, partition 0, PROCESS_LOCAL, 8427 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.541][INFO ][Executor task launch worker for task 57][Logging.scala:54]-Running task 0.0 in stage 20.0 (TID 57)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.550][INFO ][Executor task launch worker for task 57][Logging.scala:54]-Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.551][INFO ][Executor task launch worker for task 57][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.551][INFO ][Executor task launch worker for task 57][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.575][INFO ][Executor task launch worker for task 57][FileOutputCommitter.java:583]-Saved output of task 'attempt_20200102121136_0076_m_000000_0' to hdfs://192.168.98.44:8020/nari/bdp/mine/train/model/837030eeb4ea4a389366202d7daabcf6/stages/0_ClassificationPreprocess_ea5fffa313db/metadata/_temporary/0/task_20200102121136_0076_m_000000
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.575][INFO ][Executor task launch worker for task 57][Logging.scala:54]-attempt_20200102121136_0076_m_000000_0: Committed
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.576][INFO ][Executor task launch worker for task 57][Logging.scala:54]-Finished task 0.0 in stage 20.0 (TID 57). 986 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.578][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 0.0 in stage 20.0 (TID 57) in 38 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.578][INFO ][task-result-getter-1][Logging.scala:54]-Removed TaskSet 20.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.579][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 20 (runJob at SparkHadoopWriter.scala:78) finished in 0.053 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.579][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 10 finished: runJob at SparkHadoopWriter.scala:78, took 0.054098 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:36.595][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job job_20200102121136_0076 committed.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.187][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using default output committer for Parquet: parquet.hadoop.ParquetOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.224][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.224][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.226][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using user defined output committer class parquet.hadoop.ParquetOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.227][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.227][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.228][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using output committer class parquet.hadoop.ParquetOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.272][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 29.7864 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.318][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: parquet at ClassificationPreprocessModel.scala:245
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.319][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Registering RDD 79 (parquet at ClassificationPreprocessModel.scala:245)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.319][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 11 (parquet at ClassificationPreprocessModel.scala:245) with 1 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.319][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 22 (parquet at ClassificationPreprocessModel.scala:245)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.319][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List(ShuffleMapStage 21)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.319][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List(ShuffleMapStage 21)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.320][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ShuffleMapStage 21 (MapPartitionsRDD[79] at parquet at ClassificationPreprocessModel.scala:245), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.321][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_19 stored as values in memory (estimated size 5.4 KB, free 1985.2 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.323][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_19_piece0 stored as bytes in memory (estimated size 3.2 KB, free 1985.2 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.323][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Added broadcast_19_piece0 in memory on DESKTOP-ZY:49462 (size: 3.2 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.324][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 19 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.324][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ShuffleMapStage 21 (MapPartitionsRDD[79] at parquet at ClassificationPreprocessModel.scala:245) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.324][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 21.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.325][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Starting task 0.0 in stage 21.0 (TID 58, localhost, executor driver, partition 0, PROCESS_LOCAL, 8492 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.325][INFO ][Executor task launch worker for task 58][Logging.scala:54]-Running task 0.0 in stage 21.0 (TID 58)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.350][INFO ][Executor task launch worker for task 58][Logging.scala:54]-Finished task 0.0 in stage 21.0 (TID 58). 1205 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.351][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 0.0 in stage 21.0 (TID 58) in 26 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.351][INFO ][task-result-getter-3][Logging.scala:54]-Removed TaskSet 21.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.352][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ShuffleMapStage 21 (parquet at ClassificationPreprocessModel.scala:245) finished in 0.032 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.352][INFO ][dag-scheduler-event-loop][Logging.scala:54]-looking for newly runnable stages
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.352][INFO ][dag-scheduler-event-loop][Logging.scala:54]-running: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.352][INFO ][dag-scheduler-event-loop][Logging.scala:54]-waiting: Set(ResultStage 22)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.352][INFO ][dag-scheduler-event-loop][Logging.scala:54]-failed: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.353][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 22 (ShuffledRowRDD[80] at parquet at ClassificationPreprocessModel.scala:245), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.378][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_20 stored as values in memory (estimated size 153.6 KB, free 1985.0 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.380][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_20_piece0 stored as bytes in memory (estimated size 55.0 KB, free 1985.0 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.381][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Added broadcast_20_piece0 in memory on DESKTOP-ZY:49462 (size: 55.0 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.382][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 20 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.383][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ResultStage 22 (ShuffledRowRDD[80] at parquet at ClassificationPreprocessModel.scala:245) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.383][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 22.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.384][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Starting task 0.0 in stage 22.0 (TID 59, localhost, executor driver, partition 0, ANY, 7767 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.384][INFO ][Executor task launch worker for task 59][Logging.scala:54]-Running task 0.0 in stage 22.0 (TID 59)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.404][INFO ][Executor task launch worker for task 59][Logging.scala:54]-Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.404][INFO ][Executor task launch worker for task 59][Logging.scala:54]-Started 0 remote fetches in 1 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.407][INFO ][Executor task launch worker for task 59][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.407][INFO ][Executor task launch worker for task 59][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.408][INFO ][Executor task launch worker for task 59][Logging.scala:54]-Using user defined output committer class parquet.hadoop.ParquetOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.408][INFO ][Executor task launch worker for task 59][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.408][INFO ][Executor task launch worker for task 59][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.409][INFO ][Executor task launch worker for task 59][Logging.scala:54]-Using output committer class parquet.hadoop.ParquetOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.415][INFO ][Executor task launch worker for task 59][Log.java:151]-Compression: SNAPPY
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.420][INFO ][Executor task launch worker for task 59][Log.java:151]-Compression: SNAPPY
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.423][INFO ][Executor task launch worker for task 59][Log.java:151]-Parquet block size to 134217728
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.423][INFO ][Executor task launch worker for task 59][Log.java:151]-Parquet page size to 1048576
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.423][INFO ][Executor task launch worker for task 59][Log.java:151]-Parquet dictionary page size to 1048576
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.423][INFO ][Executor task launch worker for task 59][Log.java:151]-Dictionary is on
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.423][INFO ][Executor task launch worker for task 59][Log.java:151]-Validation is off
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.424][INFO ][Executor task launch worker for task 59][Log.java:151]-Writer version is: PARQUET_1_0
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.424][INFO ][Executor task launch worker for task 59][Log.java:151]-Maximum row group padding size is 8388608 bytes
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.460][INFO ][Executor task launch worker for task 59][Logging.scala:54]-Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "clsNum",
"type" : "integer",
"nullable" : false,
"metadata" : { }
}, {
"name" : "labelMap",
"type" : {
"type" : "map",
"keyType" : "double",
"valueType" : "string",
"valueContainsNull" : true
},
"nullable" : true,
"metadata" : { }
}, {
"name" : "trainColNames",
"type" : {
"type" : "array",
"elementType" : "string",
"containsNull" : true
},
"nullable" : true,
"metadata" : { }
}, {
"name" : "multiFeaInfo",
"type" : {
"type" : "map",
"keyType" : "integer",
"valueType" : "integer",
"valueContainsNull" : false
},
"nullable" : true,
"metadata" : { }
}, {
"name" : "multiFeaMap",
"type" : {
"type" : "map",
"keyType" : "string",
"valueType" : {
"type" : "map",
"keyType" : "string",
"valueType" : "double",
"valueContainsNull" : false
},
"valueContainsNull" : true
},
"nullable" : true,
"metadata" : { }
}, {
"name" : "indexerMap",
"type" : {
"type" : "map",
"keyType" : "string",
"valueType" : "string",
"valueContainsNull" : true
},
"nullable" : true,
"metadata" : { }
}, {
"name" : "clsIndexer",
"type" : {
"type" : "map",
"keyType" : "string",
"valueType" : "string",
"valueContainsNull" : true
},
"nullable" : true,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
required int32 clsNum;
optional group labelMap (MAP) {
repeated group map (MAP_KEY_VALUE) {
required double key;
optional binary value (UTF8);
}
}
optional group trainColNames (LIST) {
repeated group bag {
optional binary array (UTF8);
}
}
optional group multiFeaInfo (MAP) {
repeated group map (MAP_KEY_VALUE) {
required int32 key;
required int32 value;
}
}
optional group multiFeaMap (MAP) {
repeated group map (MAP_KEY_VALUE) {
required binary key (UTF8);
optional group value (MAP) {
repeated group map (MAP_KEY_VALUE) {
required binary key (UTF8);
required double value;
}
}
}
}
optional group indexerMap (MAP) {
repeated group map (MAP_KEY_VALUE) {
required binary key (UTF8);
optional binary value (UTF8);
}
}
optional group clsIndexer (MAP) {
repeated group map (MAP_KEY_VALUE) {
required binary key (UTF8);
optional binary value (UTF8);
}
}
}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.491][INFO ][Executor task launch worker for task 59][CodecPool.java:153]-Got brand-new compressor [.snappy]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.752][INFO ][Executor task launch worker for task 59][Log.java:151]-Flushing mem columnStore to file. allocated memory: 211
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.878][INFO ][Executor task launch worker for task 59][Log.java:151]-written 39B for [clsNum] INT32: 1 values, 4B raw, 6B comp, 1 pages, encodings: [PLAIN, BIT_PACKED]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.878][INFO ][Executor task launch worker for task 59][Log.java:151]-written 69B for [labelMap, map, key] DOUBLE: 3 values, 37B raw, 28B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.879][INFO ][Executor task launch worker for task 59][Log.java:151]-written 65B for [labelMap, map, value] BINARY: 3 values, 34B raw, 34B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.879][INFO ][Executor task launch worker for task 59][Log.java:151]-written 87B for [trainColNames, bag, array] BINARY: 4 values, 49B raw, 49B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.879][INFO ][Executor task launch worker for task 59][Log.java:151]-written 36B for [multiFeaInfo, map, key] INT32: 1 values, 13B raw, 15B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.879][INFO ][Executor task launch worker for task 59][Log.java:151]-written 36B for [multiFeaInfo, map, value] INT32: 1 values, 13B raw, 15B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.879][INFO ][Executor task launch worker for task 59][Log.java:151]-written 36B for [multiFeaMap, map, key] BINARY: 1 values, 13B raw, 15B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.879][INFO ][Executor task launch worker for task 59][Log.java:151]-written 38B for [multiFeaMap, map, value, map, key] BINARY: 1 values, 15B raw, 17B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.879][INFO ][Executor task launch worker for task 59][Log.java:151]-written 38B for [multiFeaMap, map, value, map, value] DOUBLE: 1 values, 15B raw, 17B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.880][INFO ][Executor task launch worker for task 59][Log.java:151]-written 36B for [indexerMap, map, key] BINARY: 1 values, 13B raw, 15B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.880][INFO ][Executor task launch worker for task 59][Log.java:151]-written 36B for [indexerMap, map, value] BINARY: 1 values, 13B raw, 15B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.880][INFO ][Executor task launch worker for task 59][Log.java:151]-written 192B for [clsIndexer, map, key] BINARY: 1 values, 66B raw, 67B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:37.881][INFO ][Executor task launch worker for task 59][Log.java:151]-written 94B for [clsIndexer, map, value] BINARY: 1 values, 34B raw, 35B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.108][INFO ][Executor task launch worker for task 59][FileOutputCommitter.java:583]-Saved output of task 'attempt_20200102121137_0022_m_000000_59' to hdfs://192.168.98.44:8020/nari/bdp/mine/train/model/837030eeb4ea4a389366202d7daabcf6/stages/0_ClassificationPreprocess_ea5fffa313db/data/_temporary/0/task_20200102121137_0022_m_000000
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.108][INFO ][Executor task launch worker for task 59][Logging.scala:54]-attempt_20200102121137_0022_m_000000_59: Committed
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.111][INFO ][Executor task launch worker for task 59][Logging.scala:54]-Finished task 0.0 in stage 22.0 (TID 59). 2334 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.111][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 0.0 in stage 22.0 (TID 59) in 727 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.112][INFO ][task-result-getter-0][Logging.scala:54]-Removed TaskSet 22.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.112][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 22 (parquet at ClassificationPreprocessModel.scala:245) finished in 0.759 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.113][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 11 finished: parquet at ClassificationPreprocessModel.scala:245, took 0.795299 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.129][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Write Job 1163ec8a-b4cf-4487-8513-215f8ca0957e committed.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.140][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Finished processing stats for write job 1163ec8a-b4cf-4487-8513-215f8ca0957e.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.161][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.161][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.161][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.175][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: runJob at SparkHadoopWriter.scala:78
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.176][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 12 (runJob at SparkHadoopWriter.scala:78) with 1 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.176][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 23 (runJob at SparkHadoopWriter.scala:78)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.176][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.176][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.176][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 23 (MapPartitionsRDD[83] at saveAsTextFile at ReadWrite.scala:441), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.189][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_21 stored as values in memory (estimated size 75.6 KB, free 1984.9 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.191][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_21_piece0 stored as bytes in memory (estimated size 27.3 KB, free 1984.9 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.191][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Added broadcast_21_piece0 in memory on DESKTOP-ZY:49462 (size: 27.3 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.192][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 21 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.192][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ResultStage 23 (MapPartitionsRDD[83] at saveAsTextFile at ReadWrite.scala:441) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.193][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 23.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.193][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Starting task 0.0 in stage 23.0 (TID 60, localhost, executor driver, partition 0, PROCESS_LOCAL, 8182 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.194][INFO ][Executor task launch worker for task 60][Logging.scala:54]-Running task 0.0 in stage 23.0 (TID 60)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.201][INFO ][Executor task launch worker for task 60][Logging.scala:54]-Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.201][INFO ][Executor task launch worker for task 60][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.201][INFO ][Executor task launch worker for task 60][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.226][INFO ][Executor task launch worker for task 60][FileOutputCommitter.java:583]-Saved output of task 'attempt_20200102121138_0083_m_000000_0' to hdfs://192.168.98.44:8020/nari/bdp/mine/train/model/837030eeb4ea4a389366202d7daabcf6/stages/1_stdScal_5b5b543adccf/metadata/_temporary/0/task_20200102121138_0083_m_000000
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.226][INFO ][Executor task launch worker for task 60][Logging.scala:54]-attempt_20200102121138_0083_m_000000_0: Committed
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.227][INFO ][Executor task launch worker for task 60][Logging.scala:54]-Finished task 0.0 in stage 23.0 (TID 60). 986 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.227][INFO ][task-result-getter-2][Logging.scala:54]-Finished task 0.0 in stage 23.0 (TID 60) in 34 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.227][INFO ][task-result-getter-2][Logging.scala:54]-Removed TaskSet 23.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.228][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 23 (runJob at SparkHadoopWriter.scala:78) finished in 0.051 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.228][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 12 finished: runJob at SparkHadoopWriter.scala:78, took 0.052791 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.244][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job job_20200102121138_0083 committed.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.291][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using default output committer for Parquet: parquet.hadoop.ParquetOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.292][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.292][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.292][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using user defined output committer class parquet.hadoop.ParquetOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.293][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.293][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.293][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using output committer class parquet.hadoop.ParquetOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.323][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Code generated in 21.8526 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.367][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: parquet at StandardScaler.scala:199
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.368][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Registering RDD 86 (parquet at StandardScaler.scala:199)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.368][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 13 (parquet at StandardScaler.scala:199) with 1 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.368][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 25 (parquet at StandardScaler.scala:199)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.368][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List(ShuffleMapStage 24)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.369][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List(ShuffleMapStage 24)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.369][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ShuffleMapStage 24 (MapPartitionsRDD[86] at parquet at StandardScaler.scala:199), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.372][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_22 stored as values in memory (estimated size 5.8 KB, free 1984.9 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.374][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_22_piece0 stored as bytes in memory (estimated size 3.4 KB, free 1984.9 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.374][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Added broadcast_22_piece0 in memory on DESKTOP-ZY:49462 (size: 3.4 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.375][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 22 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.375][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ShuffleMapStage 24 (MapPartitionsRDD[86] at parquet at StandardScaler.scala:199) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.375][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 24.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.376][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 0.0 in stage 24.0 (TID 61, localhost, executor driver, partition 0, PROCESS_LOCAL, 8217 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.377][INFO ][Executor task launch worker for task 61][Logging.scala:54]-Running task 0.0 in stage 24.0 (TID 61)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.401][INFO ][Executor task launch worker for task 61][Logging.scala:54]-Finished task 0.0 in stage 24.0 (TID 61). 1162 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.401][INFO ][task-result-getter-1][Logging.scala:54]-Finished task 0.0 in stage 24.0 (TID 61) in 25 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.402][INFO ][task-result-getter-1][Logging.scala:54]-Removed TaskSet 24.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.402][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ShuffleMapStage 24 (parquet at StandardScaler.scala:199) finished in 0.033 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.402][INFO ][dag-scheduler-event-loop][Logging.scala:54]-looking for newly runnable stages
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.402][INFO ][dag-scheduler-event-loop][Logging.scala:54]-running: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.402][INFO ][dag-scheduler-event-loop][Logging.scala:54]-waiting: Set(ResultStage 25)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.402][INFO ][dag-scheduler-event-loop][Logging.scala:54]-failed: Set()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.403][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 25 (ShuffledRowRDD[87] at parquet at StandardScaler.scala:199), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.429][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_23 stored as values in memory (estimated size 154.0 KB, free 1984.7 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.431][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_23_piece0 stored as bytes in memory (estimated size 55.0 KB, free 1984.7 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.432][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Added broadcast_23_piece0 in memory on DESKTOP-ZY:49462 (size: 55.0 KB, free: 1985.2 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.432][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 23 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.433][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ResultStage 25 (ShuffledRowRDD[87] at parquet at StandardScaler.scala:199) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.433][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 25.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.434][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Starting task 0.0 in stage 25.0 (TID 62, localhost, executor driver, partition 0, ANY, 7767 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.435][INFO ][Executor task launch worker for task 62][Logging.scala:54]-Running task 0.0 in stage 25.0 (TID 62)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.446][INFO ][Executor task launch worker for task 62][Logging.scala:54]-Getting 1 non-empty blocks including 1 local blocks and 0 remote blocks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.447][INFO ][Executor task launch worker for task 62][Logging.scala:54]-Started 0 remote fetches in 1 ms
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.448][INFO ][Executor task launch worker for task 62][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.449][INFO ][Executor task launch worker for task 62][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.449][INFO ][Executor task launch worker for task 62][Logging.scala:54]-Using user defined output committer class parquet.hadoop.ParquetOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.450][INFO ][Executor task launch worker for task 62][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.450][INFO ][Executor task launch worker for task 62][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.451][INFO ][Executor task launch worker for task 62][Logging.scala:54]-Using output committer class parquet.hadoop.ParquetOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.451][INFO ][Executor task launch worker for task 62][Log.java:151]-Compression: SNAPPY
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.452][INFO ][Executor task launch worker for task 62][Log.java:151]-Compression: SNAPPY
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.452][INFO ][Executor task launch worker for task 62][Log.java:151]-Parquet block size to 134217728
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.452][INFO ][Executor task launch worker for task 62][Log.java:151]-Parquet page size to 1048576
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.452][INFO ][Executor task launch worker for task 62][Log.java:151]-Parquet dictionary page size to 1048576
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.452][INFO ][Executor task launch worker for task 62][Log.java:151]-Dictionary is on
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.452][INFO ][Executor task launch worker for task 62][Log.java:151]-Validation is off
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.453][INFO ][Executor task launch worker for task 62][Log.java:151]-Writer version is: PARQUET_1_0
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.453][INFO ][Executor task launch worker for task 62][Log.java:151]-Maximum row group padding size is 8388608 bytes
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.458][INFO ][Executor task launch worker for task 62][Logging.scala:54]-Initialized Parquet WriteSupport with Catalyst schema:
{
"type" : "struct",
"fields" : [ {
"name" : "std",
"type" : {
"type" : "udt",
"class" : "org.apache.spark.ml.linalg.VectorUDT",
"pyClass" : "pyspark.ml.linalg.VectorUDT",
"sqlType" : {
"type" : "struct",
"fields" : [ {
"name" : "type",
"type" : "byte",
"nullable" : false,
"metadata" : { }
}, {
"name" : "size",
"type" : "integer",
"nullable" : true,
"metadata" : { }
}, {
"name" : "indices",
"type" : {
"type" : "array",
"elementType" : "integer",
"containsNull" : false
},
"nullable" : true,
"metadata" : { }
}, {
"name" : "values",
"type" : {
"type" : "array",
"elementType" : "double",
"containsNull" : false
},
"nullable" : true,
"metadata" : { }
} ]
}
},
"nullable" : true,
"metadata" : { }
}, {
"name" : "mean",
"type" : {
"type" : "udt",
"class" : "org.apache.spark.ml.linalg.VectorUDT",
"pyClass" : "pyspark.ml.linalg.VectorUDT",
"sqlType" : {
"type" : "struct",
"fields" : [ {
"name" : "type",
"type" : "byte",
"nullable" : false,
"metadata" : { }
}, {
"name" : "size",
"type" : "integer",
"nullable" : true,
"metadata" : { }
}, {
"name" : "indices",
"type" : {
"type" : "array",
"elementType" : "integer",
"containsNull" : false
},
"nullable" : true,
"metadata" : { }
}, {
"name" : "values",
"type" : {
"type" : "array",
"elementType" : "double",
"containsNull" : false
},
"nullable" : true,
"metadata" : { }
} ]
}
},
"nullable" : true,
"metadata" : { }
} ]
}
and corresponding Parquet message type:
message spark_schema {
optional group std {
required int32 type (INT_8);
optional int32 size;
optional group indices (LIST) {
repeated int32 array;
}
optional group values (LIST) {
repeated double array;
}
}
optional group mean {
required int32 type (INT_8);
optional int32 size;
optional group indices (LIST) {
repeated int32 array;
}
optional group values (LIST) {
repeated double array;
}
}
}
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.461][INFO ][Executor task launch worker for task 62][CodecPool.java:153]-Got brand-new compressor [.snappy]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.466][INFO ][Executor task launch worker for task 62][Log.java:151]-Flushing mem columnStore to file. allocated memory: 112
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.469][INFO ][Executor task launch worker for task 62][Log.java:151]-written 45B for [std, type] INT32: 1 values, 10B raw, 12B comp, 1 pages, encodings: [PLAIN, BIT_PACKED, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.469][INFO ][Executor task launch worker for task 62][Log.java:151]-written 30B for [std, size] INT32: 1 values, 7B raw, 9B comp, 1 pages, encodings: [PLAIN, BIT_PACKED, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.470][INFO ][Executor task launch worker for task 62][Log.java:151]-written 36B for [std, indices, array] INT32: 1 values, 13B raw, 15B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.471][INFO ][Executor task launch worker for task 62][Log.java:151]-written 87B for [std, values, array] DOUBLE: 4 values, 45B raw, 46B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.471][INFO ][Executor task launch worker for task 62][Log.java:151]-written 45B for [mean, type] INT32: 1 values, 10B raw, 12B comp, 1 pages, encodings: [PLAIN, BIT_PACKED, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.471][INFO ][Executor task launch worker for task 62][Log.java:151]-written 30B for [mean, size] INT32: 1 values, 7B raw, 9B comp, 1 pages, encodings: [PLAIN, BIT_PACKED, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.472][INFO ][Executor task launch worker for task 62][Log.java:151]-written 36B for [mean, indices, array] INT32: 1 values, 13B raw, 15B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.472][INFO ][Executor task launch worker for task 62][Log.java:151]-written 87B for [mean, values, array] DOUBLE: 4 values, 45B raw, 46B comp, 1 pages, encodings: [PLAIN, RLE]
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.523][INFO ][Executor task launch worker for task 62][FileOutputCommitter.java:583]-Saved output of task 'attempt_20200102121138_0025_m_000000_62' to hdfs://192.168.98.44:8020/nari/bdp/mine/train/model/837030eeb4ea4a389366202d7daabcf6/stages/1_stdScal_5b5b543adccf/data/_temporary/0/task_20200102121138_0025_m_000000
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.523][INFO ][Executor task launch worker for task 62][Logging.scala:54]-attempt_20200102121138_0025_m_000000_62: Committed
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.525][INFO ][Executor task launch worker for task 62][Logging.scala:54]-Finished task 0.0 in stage 25.0 (TID 62). 2248 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.526][INFO ][task-result-getter-3][Logging.scala:54]-Finished task 0.0 in stage 25.0 (TID 62) in 91 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.526][INFO ][task-result-getter-3][Logging.scala:54]-Removed TaskSet 25.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.527][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 25 (parquet at StandardScaler.scala:199) finished in 0.124 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.528][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 13 finished: parquet at StandardScaler.scala:199, took 0.159800 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.550][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Write Job 22fe0647-5413-4b23-9795-aee661a03bcb committed.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.551][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Finished processing stats for write job 22fe0647-5413-4b23-9795-aee661a03bcb.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.570][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.570][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.571][INFO ][http-nio-20779-exec-1][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.587][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Starting job: runJob at SparkHadoopWriter.scala:78
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.587][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Got job 14 (runJob at SparkHadoopWriter.scala:78) with 1 output partitions
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.588][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Final stage: ResultStage 26 (runJob at SparkHadoopWriter.scala:78)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.588][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Parents of final stage: List()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.588][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Missing parents: List()
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.588][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting ResultStage 26 (MapPartitionsRDD[90] at saveAsTextFile at ReadWrite.scala:441), which has no missing parents
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.602][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_24 stored as values in memory (estimated size 75.6 KB, free 1984.6 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.604][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Block broadcast_24_piece0 stored as bytes in memory (estimated size 27.3 KB, free 1984.6 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.605][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Added broadcast_24_piece0 in memory on DESKTOP-ZY:49462 (size: 27.3 KB, free: 1985.2 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.605][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Created broadcast 24 from broadcast at DAGScheduler.scala:1161
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.606][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Submitting 1 missing tasks from ResultStage 26 (MapPartitionsRDD[90] at saveAsTextFile at ReadWrite.scala:441) (first 15 tasks are for partitions Vector(0))
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.606][INFO ][dag-scheduler-event-loop][Logging.scala:54]-Adding task set 26.0 with 1 tasks
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.607][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Starting task 0.0 in stage 26.0 (TID 63, localhost, executor driver, partition 0, PROCESS_LOCAL, 8893 bytes)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.607][INFO ][Executor task launch worker for task 63][Logging.scala:54]-Running task 0.0 in stage 26.0 (TID 63)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.615][INFO ][Executor task launch worker for task 63][Logging.scala:54]-Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.615][INFO ][Executor task launch worker for task 63][FileOutputCommitter.java:124]-File Output Committer Algorithm version is 1
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.615][INFO ][Executor task launch worker for task 63][FileOutputCommitter.java:139]-FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.643][INFO ][Executor task launch worker for task 63][FileOutputCommitter.java:583]-Saved output of task 'attempt_20200102121138_0090_m_000000_0' to hdfs://192.168.98.44:8020/nari/bdp/mine/train/model/837030eeb4ea4a389366202d7daabcf6/stages/2_DNNClassification_86c02da7d36f/metadata/_temporary/0/task_20200102121138_0090_m_000000
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.643][INFO ][Executor task launch worker for task 63][Logging.scala:54]-attempt_20200102121138_0090_m_000000_0: Committed
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.644][INFO ][Executor task launch worker for task 63][Logging.scala:54]-Finished task 0.0 in stage 26.0 (TID 63). 986 bytes result sent to driver
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.645][INFO ][task-result-getter-0][Logging.scala:54]-Finished task 0.0 in stage 26.0 (TID 63) in 38 ms on localhost (executor driver) (1/1)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.645][INFO ][task-result-getter-0][Logging.scala:54]-Removed TaskSet 26.0, whose tasks have all completed, from pool
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.645][INFO ][dag-scheduler-event-loop][Logging.scala:54]-ResultStage 26 (runJob at SparkHadoopWriter.scala:78) finished in 0.056 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.646][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job 14 finished: runJob at SparkHadoopWriter.scala:78, took 0.058776 s
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.661][INFO ][http-nio-20779-exec-1][Logging.scala:54]-Job job_20200102121138_0090 committed.
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:38.819][ERROR][http-nio-20779-exec-1][DirectJDKLog.java:175]-Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.reflect.UndeclaredThrowableException] with root cause
scala.reflect.internal.Symbols$CyclicReference: illegal cyclic reference involving type T
at scala.reflect.internal.Symbols$Symbol$$anonfun$info$3.apply(Symbols.scala:1523)
at scala.reflect.internal.Symbols$Symbol$$anonfun$info$3.apply(Symbols.scala:1521)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.reflect.internal.Symbols$Symbol.lock(Symbols.scala:567)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1521)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$11.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:159)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$11.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:159)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$11.info(SynchronizedSymbols.scala:159)
at scala.reflect.internal.Types$$anonfun$32.apply(Types.scala:3784)
at scala.reflect.internal.Types$$anonfun$32.apply(Types.scala:3783)
at scala.reflect.internal.util.Collections$class.mapWithIndex(Collections.scala:193)
at scala.reflect.internal.SymbolTable.mapWithIndex(SymbolTable.scala:16)
at scala.reflect.internal.Types$class.typeParamsToExistentials(Types.scala:3783)
at scala.reflect.internal.SymbolTable.typeParamsToExistentials(SymbolTable.scala:16)
at scala.reflect.internal.Types$class.typeParamsToExistentials(Types.scala:3789)
at scala.reflect.internal.SymbolTable.typeParamsToExistentials(SymbolTable.scala:16)
at scala.reflect.internal.tpe.TypeMaps$$anon$1.apply(TypeMaps.scala:332)
at scala.reflect.runtime.JavaMirrors$JavaMirror.typeToScala(JavaMirrors.scala:1086)
at scala.reflect.runtime.JavaMirrors$JavaMirror$TypeParamCompleter$$anonfun$complete$1.apply(JavaMirrors.scala:665)
at scala.reflect.runtime.JavaMirrors$JavaMirror$TypeParamCompleter$$anonfun$complete$1.apply(JavaMirrors.scala:665)
at scala.collection.immutable.List.map(List.scala:284)
at scala.reflect.runtime.JavaMirrors$JavaMirror$TypeParamCompleter.complete(JavaMirrors.scala:665)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1535)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$11.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:159)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$11.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:159)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$11.info(SynchronizedSymbols.scala:159)
at scala.reflect.internal.Types$$anonfun$32.apply(Types.scala:3784)
at scala.reflect.internal.Types$$anonfun$32.apply(Types.scala:3783)
at scala.reflect.internal.util.Collections$class.mapWithIndex(Collections.scala:193)
at scala.reflect.internal.SymbolTable.mapWithIndex(SymbolTable.scala:16)
at scala.reflect.internal.Types$class.typeParamsToExistentials(Types.scala:3783)
at scala.reflect.internal.SymbolTable.typeParamsToExistentials(SymbolTable.scala:16)
at scala.reflect.internal.Types$class.typeParamsToExistentials(Types.scala:3789)
at scala.reflect.internal.SymbolTable.typeParamsToExistentials(SymbolTable.scala:16)
at scala.reflect.internal.tpe.TypeMaps$$anon$1.apply(TypeMaps.scala:332)
at scala.reflect.runtime.JavaMirrors$JavaMirror.typeToScala(JavaMirrors.scala:1086)
at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$22.apply(JavaMirrors.scala:1158)
at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$22.apply(JavaMirrors.scala:1158)
at scala.collection.immutable.List.map(List.scala:288)
at scala.reflect.runtime.JavaMirrors$JavaMirror.scala$reflect$runtime$JavaMirrors$JavaMirror$$jmethodAsScala1(JavaMirrors.scala:1158)
at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$scala$reflect$runtime$JavaMirrors$JavaMirror$$jmethodAsScala$1.apply(JavaMirrors.scala:1151)
at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$scala$reflect$runtime$JavaMirrors$JavaMirror$$jmethodAsScala$1.apply(JavaMirrors.scala:1151)
at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$toScala$1.apply(JavaMirrors.scala:97)
at scala.reflect.runtime.TwoWayCaches$TwoWayCache$$anonfun$toScala$1.apply(TwoWayCaches.scala:38)
at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
at scala.reflect.runtime.TwoWayCaches$TwoWayCache.toScala(TwoWayCaches.scala:33)
at scala.reflect.runtime.JavaMirrors$JavaMirror.toScala(JavaMirrors.scala:95)
at scala.reflect.runtime.JavaMirrors$JavaMirror.scala$reflect$runtime$JavaMirrors$JavaMirror$$jmethodAsScala(JavaMirrors.scala:1151)
at scala.reflect.runtime.JavaMirrors$JavaMirror$FromJavaClassCompleter$$anonfun$completeRest$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply(JavaMirrors.scala:782)
at scala.reflect.runtime.JavaMirrors$JavaMirror$FromJavaClassCompleter$$anonfun$completeRest$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply(JavaMirrors.scala:782)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at scala.reflect.runtime.JavaMirrors$JavaMirror$FromJavaClassCompleter$$anonfun$completeRest$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(JavaMirrors.scala:782)
at scala.reflect.runtime.JavaMirrors$JavaMirror$FromJavaClassCompleter$$anonfun$completeRest$1.apply$mcV$sp(JavaMirrors.scala:791)
at scala.reflect.runtime.JavaMirrors$JavaMirror$FromJavaClassCompleter$$anonfun$completeRest$1.apply(JavaMirrors.scala:749)
at scala.reflect.runtime.JavaMirrors$JavaMirror$FromJavaClassCompleter$$anonfun$completeRest$1.apply(JavaMirrors.scala:749)
at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
at scala.reflect.runtime.JavaMirrors$JavaMirror$FromJavaClassCompleter.completeRest(JavaMirrors.scala:749)
at scala.reflect.runtime.JavaMirrors$JavaMirror$FromJavaClassCompleter$LazyPolyType.complete(JavaMirrors.scala:798)
at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1535)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$8.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:168)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$8.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:168)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$8.info(SynchronizedSymbols.scala:168)
at scala.reflect.internal.Symbols$Symbol.isDerivedValueClass(Symbols.scala:824)
at scala.reflect.internal.transform.Erasure$ErasureMap.applyInArray(Erasure.scala:160)
at scala.reflect.internal.transform.Erasure$ErasureMap$$anonfun$apply$1.apply(Erasure.scala:127)
at scala.reflect.internal.transform.Erasure$ErasureMap$$anonfun$apply$1.apply(Erasure.scala:127)
at scala.collection.immutable.List.map(List.scala:284)
at scala.reflect.internal.transform.Erasure$ErasureMap.apply(Erasure.scala:127)
at scala.reflect.internal.transform.Transforms$class.transformedType(Transforms.scala:43)
at scala.reflect.internal.SymbolTable.transformedType(SymbolTable.scala:16)
at scala.reflect.internal.Types$TypeApiImpl.erasure(Types.scala:225)
at scala.reflect.internal.Types$TypeApiImpl.erasure(Types.scala:218)
at org.apache.spark.sql.catalyst.ScalaReflection$class.getClassNameFromType(ScalaReflection.scala:935)
at org.apache.spark.sql.catalyst.ScalaReflection$.getClassNameFromType(ScalaReflection.scala:46)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:731)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:724)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:906)
at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:46)
at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:723)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1$$anonfun$apply$8.apply(ScalaReflection.scala:785)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1$$anonfun$apply$8.apply(ScalaReflection.scala:784)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.immutable.List.map(List.scala:296)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:784)
at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:724)
at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:906)
at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:46)
at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:723)
at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:720)
at org.apache.spark.sql.SparkSession.createDataFrame(SparkSession.scala:313)
at org.apache.spark.ml.nari.algorithm.classification.dnn.DNNClassificationModel$DNNClassificationModelWriter.saveImpl(DNNClassificationModel.scala:174)
at org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:180)
at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$saveImpl$1.apply(Pipeline.scala:254)
at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$saveImpl$1.apply(Pipeline.scala:253)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at org.apache.spark.ml.Pipeline$SharedReadWrite$.saveImpl(Pipeline.scala:253)
at org.apache.spark.ml.PipelineModel$PipelineModelWriter.saveImpl(Pipeline.scala:338)
at org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:180)
at org.apache.spark.ml.util.MLWritable$class.save(ReadWrite.scala:306)
at org.apache.spark.ml.PipelineModel.save(Pipeline.scala:293)
at com.nari.bdp.commons.mine.base.behaviors.BaseBehavior.saveModel(BaseBehavior.java:236)
at org.apache.spark.ml.nari.algorithm.classification.dnn.behavior.DnnClassificationBehavior.execute(DnnClassificationBehavior.scala:168)
at com.nari.bdp.mine.dl.classification.dnn.core.service.AsyncServiceImpl.arimaAsync(AsyncServiceImpl.java:20)
at com.nari.bdp.mine.dl.classification.dnn.core.service.DnnService.dnnWithFlow(DnnService.java:68)
at com.nari.bdp.mine.dl.classification.dnn.controller.DnnClassificationController.excute(DnnClassificationController.java:51)
at com.nari.bdp.mine.dl.classification.dnn.controller.DnnClassificationController$$FastClassBySpringCGLIB$$da72b424.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
at com.nari.bdp.mine.dl.classification.dnn.controller.DnnClassificationController$$EnhancerBySpringCGLIB$$41ab79cf.excute(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1579)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.892][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 698
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.892][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 689
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.892][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 644
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.892][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 647
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.892][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 400
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.892][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 548
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.892][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 615
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.893][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 347
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.893][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 635
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.893][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 629
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.893][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 568
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.893][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 657
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.893][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 493
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.893][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 620
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.893][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 606
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.893][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 681
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.894][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 569
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.894][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 558
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.894][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 584
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.894][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 556
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.894][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 663
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.894][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 549
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.894][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 580
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.894][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 597
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.896][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Removed broadcast_21_piece0 on DESKTOP-ZY:49462 in memory (size: 27.3 KB, free: 1985.2 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 649
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 671
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 528
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 636
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 540
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 559
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 680
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.898][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 542
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.899][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 531
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.899][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned shuffle 9
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.899][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 578
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.899][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 637
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 611
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 633
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 694
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 613
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.900][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 648
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 593
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 518
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 642
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 592
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 529
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 534
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 664
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.901][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 685
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 585
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 652
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 625
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 530
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 599
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 502
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 526
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 670
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 673
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.902][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 654
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 574
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 651
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 662
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 494
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 560
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 496
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 665
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 538
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.903][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 653
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.904][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 522
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.904][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 525
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.904][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 543
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.904][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 582
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.904][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 595
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.904][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 628
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.904][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 591
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.906][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Removed broadcast_22_piece0 on DESKTOP-ZY:49462 in memory (size: 3.4 KB, free: 1985.2 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.908][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 701
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.909][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 547
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.909][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 524
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.909][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 616
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.909][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 634
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.910][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Removed broadcast_24_piece0 on DESKTOP-ZY:49462 in memory (size: 27.3 KB, free: 1985.2 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.912][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 598
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.912][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 571
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.913][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 521
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.913][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 610
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.913][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 535
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.913][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 630
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.913][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 658
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.913][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 564
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.913][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned shuffle 6
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.916][INFO ][dispatcher-event-loop-2][Logging.scala:54]-Removed broadcast_23_piece0 on DESKTOP-ZY:49462 in memory (size: 55.0 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.920][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 608
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.920][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 581
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 679
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 623
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 656
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 504
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 703
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 640
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 618
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 544
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 344
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 491
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 563
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 519
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 695
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 492
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 676
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 567
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 655
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 495
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 614
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 632
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.921][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 570
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 686
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 678
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 552
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 602
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 555
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 659
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 696
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 576
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 554
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 645
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 666
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 661
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 631
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 639
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 607
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.922][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 500
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.924][INFO ][dispatcher-event-loop-3][Logging.scala:54]-Removed broadcast_20_piece0 on DESKTOP-ZY:49462 in memory (size: 55.0 KB, free: 1985.3 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.926][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Removed broadcast_18_piece0 on DESKTOP-ZY:49462 in memory (size: 27.3 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.928][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 514
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.928][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 533
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.928][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 507
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.928][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 683
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.928][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 523
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.929][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 503
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.929][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 650
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.929][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 501
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.929][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 579
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.929][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 682
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.929][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 527
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.929][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 341
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.929][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 553
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.929][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 690
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.930][INFO ][dispatcher-event-loop-1][Logging.scala:54]-Removed broadcast_19_piece0 on DESKTOP-ZY:49462 in memory (size: 3.2 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.932][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 511
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.932][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 515
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.932][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 605
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.932][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 586
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.932][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 641
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.933][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 621
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.933][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 509
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.933][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 561
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.933][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 513
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.933][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 575
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.933][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 622
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.933][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 684
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.933][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 550
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.933][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 532
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.933][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 674
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.933][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 626
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 672
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 638
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 498
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 546
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 510
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 516
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 702
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 346
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 643
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 688
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 609
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.934][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 499
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 577
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 675
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 512
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 562
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 646
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 604
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 700
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 572
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 551
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 617
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 687
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.935][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 345
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.936][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 566
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.936][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 619
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.936][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 536
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.936][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 594
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.936][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 692
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.936][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 497
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.936][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 505
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.936][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 624
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.937][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned shuffle 8
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.937][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 612
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.937][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 587
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.937][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 693
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.937][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 506
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.937][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 520
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.937][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 541
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.939][INFO ][dispatcher-event-loop-0][Logging.scala:54]-Removed broadcast_17_piece0 on DESKTOP-ZY:49462 in memory (size: 27.2 KB, free: 1985.4 MB)
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.942][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 539
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.942][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 573
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.942][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 583
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.942][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 600
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.942][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 669
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.942][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 517
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.942][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 699
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 537
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 668
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 557
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 660
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 691
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 603
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 545
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 588
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 601
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 565
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 697
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.943][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 627
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.944][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 589
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.944][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 596
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.944][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 667
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.944][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 704
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.944][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 677
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.944][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 508
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:11:39.944][INFO ][Spark Context Cleaner][Logging.scala:54]-Cleaned accumulator 590
[lf-1][bdp-mine-dl-classification-dnn][2020-01-02 12:15:25.115][INFO ][AsyncResolver-bootstrap-executor-0][ConfigClusterResolver.java:43]-Resolving eureka endpoints via configuration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment