GVKun编程网logo

Ubuntu 16.04后台运行scrapy爬虫程序(ubuntu怎么跑python脚本)

1

对于Ubuntu16.04后台运行scrapy爬虫程序感兴趣的读者,本文将提供您所需要的所有信息,我们将详细讲解ubuntu怎么跑python脚本,并且为您提供关于apt-getupdate失败ubu

对于Ubuntu 16.04后台运行scrapy爬虫程序感兴趣的读者,本文将提供您所需要的所有信息,我们将详细讲解ubuntu怎么跑python脚本,并且为您提供关于apt-get update 失败 ubuntu:Tempory failure resolving 'cn.archive.ubuntu.com ubuntu、Debian 9 / Debian 10 / Ubuntu 18.04 / Ubuntu 18.10 快速开启 BBR 加速 或 关闭 BBR 加速、install nginx on ubuntu install ubuntu usb install ubuntu 14.04 ubuntu install jd、Intel RealSense D435i Installation on Linux (Ubuntu 16.04 and Ubuntu 18.04)的宝贵知识。

本文目录一览:

Ubuntu 16.04后台运行scrapy爬虫程序(ubuntu怎么跑python脚本)

Ubuntu 16.04后台运行scrapy爬虫程序(ubuntu怎么跑python脚本)

某些爬虫程序需要运行很长时间才能将数据爬完,爬取太快呢又会被网站给封禁。你又不想一直开着电脑连续开几天,太麻烦。。。

其实有个好方法,你可以把爬虫放在阿里云服务器运行,这样你就不需要管了,但是你如果在Ubuntu或阿里云上直接:

scrapy crawl spider_name 或python run.py的话

当你关闭链接阿里云的xshell时,程序会直接停掉不会继续运行。

今天给大家分享一个在阿里云服务器后台运行你的scrapy爬虫代码的命令,可以使你的爬虫在服务器后台一直运行,关闭连接也没事。

nohup python -u run.py > spider_house.log 2>&1 &

run.py为你自己写的scrapy爬虫的运行文件:

from scrapy.cmdline import execute
execute(['scrapy','crawl','house'])

执行完命令后会返回给你一个进程ID,此时你的爬虫就已经在该进程中运行了,你可以用下边命令查看后台该进程:

ps -aux   #a:显示所有程序 u:以用户为主的格式来显示 x:显示所有程序,不以终端机来区分

只要找到返回给你的进程pid,说明你的爬虫正在运行。
如下是我的运行进程:

apt-get update 失败 ubuntu:Tempory failure resolving 'cn.archive.ubuntu.com ubuntu

apt-get update 失败 ubuntu:Tempory failure resolving 'cn.archive.ubuntu.com ubuntu

当运行apt-get update后出现如下错误时:
E: Some index files Failed to download,they have been ignored,or old ones used instead.

可以将目录下/var/lib/apt/lists/partial/所有的文件清掉,再次运行apt-get update即可!自带源在大陆不好。


出现以下错误:

ubuntu:Tempory failure resolving ''cn.archive.ubuntu.com ubuntu


修改dns:

1,重启生效:

sudovi/etc/resolvconf/resolv.conf.d/base(这个文件默认是空的)

在里面插入:
nameserver8.8.8.8
nameserver8.8.4.4

如果有多个DNS就一行一个

修改好保存,然后执行

sudoresolvconf-u

再看/etc/resolv.conf,最下面就多了2行:

cat/etc/resolv.conf
#Dynamicresolv.conf(5)fileforglibcresolver(3)generatedbyresolvconf(8)
#DONOTEDITTHISFILEBYHAND--YOURCHANGESWILLBEOVERWRITTEN
可以看到我们的设置已经加上了,然后再ping一个域名,当时就可以解析了,无需重启。


2,重启失效:

配置文件地址 /etc/resolv.conf

使用编辑器打开

改为如下内容:
search localdomain
nameserver 202.96.128.86 希望修改成的DNS
nameserver 202.96.128.166 备用DNS

重启网络:sudo /etc/init.d/networking restart。即可

Debian 9 / Debian 10 / Ubuntu 18.04 / Ubuntu 18.10 快速开启 BBR 加速 或 关闭 BBR 加速

Debian 9 / Debian 10 / Ubuntu 18.04 / Ubuntu 18.10 快速开启 BBR 加速 或 关闭 BBR 加速

如果使用的是 Debian 9、Debian 10、Ubuntu 18.04、Ubuntu 18.10 等内核高于 4.9 版本的系统,均可以使用此方法开启 BBR 加速,若你使用了 Ubuntu 19.04 的系统无需开启,系统默认就开启了。虽然 BBR 没有锐速那么暴力,但是兼容性和稳定性占优势,推荐大家使用。对与什么是 BBR 我就不详细说明了,这是一款由谷歌推出的 TCP 单边加速的拥塞控制算法。

 

开启 BBR

1. 修改 sysctl.conf 系统参数

echo "net.core.default_qdisc=fq" >> /etc/sysctl.conf
echo "net.ipv4.tcp_congestion_control=bbr" >> /etc/sysctl.conf

 

2. 保存并生效

sysctl -p

得到返回值如下:(最后两行有就可以了)

 

3. 查看是否已开启 BBR

sysctl net.ipv4.tcp_available_congestion_control

若得到返回值带有 bbr 则成功开启,如:

 

4. 检查 BBR 是否成功启动

lsmod | grep bbr

如返回信息则表示已经成功开启:

 

如果不确定自己的系统是支持快速开启 bbr 可以查看使用下面的命令查看内核,只要 4.9 及以上版本均可直接开启:

uname -r

 

关闭 BBR

若想要关闭 bbr 加速也很简单,只需要将第二步的返回内容在 sysctl.conf 内删除或注释掉即可。

 

1. 修改配置文件

vi /etc/sysctl.conf

进入配置文件,将添加的内容在最后,若找不到往下翻,按 i 键或 Insert 键开启编辑,注释或删除内容后,按 ESC 键退出编辑,键入

:wq

保存并退出。

 

2. 保存配置

sysctl -p

 

3. 重启系统生效

reboot

 

3. 查看是否已关闭 BBR

sysctl net.ipv4.tcp_available_congestion_control

 

 

install nginx on ubuntu install ubuntu usb install ubuntu 14.04 ubuntu install jd

install nginx on ubuntu install ubuntu usb install ubuntu 14.04 ubuntu install jd

Intel RealSense D435i Installation on Linux (Ubuntu 16.04 and Ubuntu 18.04)

Intel RealSense D435i Installation on Linux (Ubuntu 16.04 and Ubuntu 18.04)

1. Install 3rd-party dependencies

1.1 apt-get update

sudo apt-get update
  • 1

1.2 install libusb-1.0, libglfw, freeglut,

sudo apt-get install libusb-dev libusb-1.0-0-dev libglfw3 libglfw3-dev freeglut3 freeglut3-dev
  • 1

1.3 Install libpng12-dev

1.3.1 On Ubuntu 16.04, run the following command:

sudo apt-get install libpng12-dev

1.3.2 On Ubuntu 18.04, run the following command:

wget -q -O /tmp/libpng12.deb http://security.ubuntu.com/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.54-1ubuntu1.1_amd64.deb

sudo dpkg -i /tmp/libpng12.deb

2. Debian Package Installation

2.1 Add Intel server to the list of repositories :

echo 'deb http://realsense-hw-public.s3.amazonaws.com/Debian/apt-repo xenial main' | sudo tee /etc/apt/sources.list.d/realsense-public.list
  • 1

It is recommended to backup /etc/apt/sources.list.d/realsense-public.list file in case of an upgrade.

2.2 Register the server’s public key :

sudo apt-key adv --keyserver keys.gnupg.net --recv-key 6F3EFCDE
sudo apt-key adv --keyserver keys.gnupg.net --recv-key C8B3A55A6F3EFCDE || sudo apt-key adv -- keyserver hkp://keyserver.ubuntu.com:80 --recv-key C8B3A55A6F3EFCDE
  • 1
  • 2

Refresh the list of repositories and packages available :

sudo apt-get update
  • 1

2.3 Install the librscalibrationtool package which includes Intel® RealSense™ Dynamic Calibrator:

sudo apt-get install librscalibrationtool
  • 1

说明:

librscalibrationtool适用设备为 Intel RealSense D400, D410, D415, D420, D430, D435.
安装后的使用说明见: /usr/share/doc/librscalibrationtool/README.md

2.4 Failure – (Optional) Developers shall also install librscalibrationapi package:

sudo apt-get install librscalibrationapi
  • 1

Problem:

librscalibrationapi : Depends: libpng12-dev but it is not installable
分析: 自Ubuntu18.04以后,libpng12-dev已经被弃用

3. Checking Package Installation

After the debian package install finishes, check for files installed. Under debian convention, the executables are installed under /usr/bin, library files under /usr/lib, and other files including sample code under /usr/share/doc.
For example, for Calibration Tool, librscalibrationtool, the files are installed as below:

sudo dpkg -L librscalibrationtool
  • 1

4. Video4Linux backend

  1. In order to run demos install:
sudo apt-get install librealsense2-dkms
sudo apt-get install librealsense2-utils
  • 1
  • 2

The above two lines will deploy librealsense2 udev rules, kernel drivers, runtime library and executable demos and tools.
Reconnect the Intel RealSense depth camera and run: realsense-viewer

  1. Developers shall install additional packages:
sudo apt-get install librealsense2-dev
sudo apt-get install librealsense2-dbg
  • 1
  • 2

With dev package installed, you can compile an application with librealsense using g++ -std=c++11 filename.cpp -lrealsense2 or an IDE of your choice.

  1. Verify that the kernel is updated :
modinfo uvcvideo | grep "version:" 
  • 1

should include realsense string

5. Test Installation

  1. Print the tool version to screen:
/usr/bin/Intel.Realsense.DynamicCalibrator -v
  • 1
  1. To print device information to screen, connect a camera device to the system and execute the following command:
/usr/bin/Intel.Realsense.DynamicCalibrator -list
  • 1

6. Perform Calibration for Camera

/usr/bin/Intel.Realsense.DynamicCalibrator
  • 1

* Problem (not fixed):

I can see the RGB, Depth and Gyro information without problems when I run realsense-vieweror subscribe to the realsense-ros topics. But when I run/usr/bin/Intel.Realsense.DynamicCalibrator
And click on Start Calibration I get the Failed to start Calibration Error as seen:

PS: relative issue discussion on github:
https://github.com/IntelRealSense/librealsense/issues/3990

关于Ubuntu 16.04后台运行scrapy爬虫程序ubuntu怎么跑python脚本的问题就给大家分享到这里,感谢你花时间阅读本站内容,更多关于apt-get update 失败 ubuntu:Tempory failure resolving 'cn.archive.ubuntu.com ubuntu、Debian 9 / Debian 10 / Ubuntu 18.04 / Ubuntu 18.10 快速开启 BBR 加速 或 关闭 BBR 加速、install nginx on ubuntu install ubuntu usb install ubuntu 14.04 ubuntu install jd、Intel RealSense D435i Installation on Linux (Ubuntu 16.04 and Ubuntu 18.04)等相关知识的信息别忘了在本站进行查找喔。

本文标签:

上一篇ubuntu 设置root用户密码并实现root用户登录(ubuntu 设置root账号)

下一篇Ubuntu server 18.04 anaconda 安装+配置系统环境变量(ubuntu安装anaconda3环境变量)