These are the steps I took to install twint and get it to work!
Just a quick update on how to install twint as I was able to download,
install && run a search using Google Cloud Shell. For
instructions, check below for No3 to get right into it. After installing, I closed the window and started up the Shell again and ran the same commands for twint, and it didn't work for me. I got a token error. But i can't find a fix for it.
What you will need to install beforehand are.
- apt install git
- apt install python3
- apt install python3-pip
- Google Cloud Account
1: Install on Kali. This did not work for me.
Linux Kali-ROG 5.14.0-kali4-amd64 #1 SMP Debian 5.14.16-1kali1 (2021-11-05) x86_64 GNU/Linux
Instructions for install : https://github.com/twintproject/twint
- git clone –depth=1 https://github.com/twintproject/twint.git
- cd twint
- pip3 install . -r requirements.txt
Try and run these commands to see if it works.
$ twint -s pineapple
$ twint -u networkchuck -s "raspberry pi"
2: Install on Kali AWS. This did not work for me.
Linux kali 5.15.0-kali2-cloud-amd64 #1 SMP Debian 5.15.5-2kali2 (2021-12-22) x86_64 GNU/Linux
It seems that twint and AWS do not work due to api keys.
Instructions for install : https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/
- sudo git clone –depth=1 https://github.com/twintproject/twint.git
- sudo cd twint
- sudo pip3 install . -r requirements.txt
- sudo pip3 install twint
Try and run these commands to see if it works.
$ twint -s pineapple
$ twint -u networkchuck -s "raspberry pi"
3: Install on Google Cloud Shell and This worked for me.
Linux cs-899333161534-default-boost-h8jtm 5.10.68+ #1 SMP Wed Dec 1 10:07:21 UTC 2021 x86_64 GNU/Linux
Instructions for install : https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/
FYI 1 - Google didn't like the command git clone --depth=1
So I removed it and it installed.
FY 2 - Using pip3 install twint command said all was installed, but I added sudo
beforehand and it actually installed it seems.
$ sudo git clone https://github.com/twintproject/twint.git
$ ls
$ cd twint
$ ls
$ sudo pip3 install . -r requirements.txt
$ sudo pip3 install twint
Try and run these commands to see if it works.
$ twint -s pineapple
$ twint -u networkchuck -s "raspberry pi"
If you see these results then it worked for you also.
The Kali twint install journey.
I have had twint installed before but that build was deleted as I have started a fresh install of Kali now working really nice.
Remember to backup:
I use GParted on a USB_key and external HDD to savethe backup).
After 5 hours of research, 5 long hours of holding back tears ;) to try and get the package twint to word for me.
To
say if your looking for which way worked for me, it was number 3 on the
list using Googles Cloud Shell. I had to install the package,
requirements and another step that some sites and even the main GitHub
page didn't have, that was the step to install twint. See No 2 & No
3.
As a complete noob, using No 1 from Github instructions
may work on some other version of Linux, but not mine and it also seems
Amazon's AWS version of Kali-Linux either.
My 1st impulse is to blame my lack of knowledge for the problem.
My 2nd impulse is to think an update may have stopped this from working today, but will be ok tomorrow. Like with NordVPN two months ago.Install Steps made on Kali Linux which did not work on two machines listed above.
FYI - Attempts to fix the problem will be below all the setps.
My Install.
┌──(kali㉿kali)-[~]
└─$ git clone --depth=1 https://github.com/twintproject/twint.git
Cloning into 'twint'...
remote: Enumerating objects: 47, done.
remote: Counting objects: 100% (47/47), done.
remote: Compressing objects: 100% (44/44), done.
remote: Total 47 (delta 3), reused 14 (delta 0), pack-reused 0
Receiving objects: 100% (47/47), 42.95 KiB | 1.59 MiB/s, done.
Resolving deltas: 100% (3/3), done.
┌──(kali㉿kali)-[~]
└─$ ls
Desktop Documents Downloads Music Pictures Public Templates Videos bash.txt nmap twint
┌──(kali㉿kali)-[~]
└─$ cd twint
┌──(kali㉿kali)-[~/twint]
└─$cd twint
┌──(kali㉿kali)-[~/twint]
└─$ pip3 install . -r requirements.txt
Command 'pip3' not found, but can be installed with:
sudo apt install python3-pip
Do you want to install it? (N/y)y
sudo apt install python3-pip
┌──(kali㉿kali)-[~/twint]
└─$ pip3 install . -r requirements.txt
WARNING: The script twint is installed in '/home/kali/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed aiohttp-socks-0.4.1 cchardet-2.1.7 dataclasses-0.6 elasticsearch-7.16.2 fake-useragent-0.1.11 geographiclib-1.52 geopy-2.2.0 googletransx-2.4.2 pandas-1.3.5 schedule-1.1.0 twint-2.1.21
(I added two commands for the path later on in the install, see far below).
┌──(kali㉿kali)-[~/.local]
└─$ cd bin
┌──(kali㉿kali)-[~/.local/bin]
└─$ ls
cchardetect translate twint
┌──(kali㉿kali)-[~/.local/bin]
└─$ twint -u networkchuck
Traceback (most recent call last):
File "/usr/local/bin/twint", line 33, in <module>
sys.exit(load_entry_point('twint==2.1.21', 'console_scripts', 'twint')())
File "/usr/local/bin/twint", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/home/kali/.local/lib/python3.9/site-packages/twint/__init__.py", line 14, in <module>
from . import run
File "/home/kali/.local/lib/python3.9/site-packages/twint/run.py", line 4, in <module>
from . import datelock, feed, get, output, verbose, storage
File "/home/kali/.local/lib/python3.9/site-packages/twint/get.py", line 12, in <module>
from aiohttp_socks import ProxyConnector, ProxyType
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/__init__.py", line 5, in <module>
from .connector import (
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/connector.py", line 8, in <module>
from aiohttp.helpers import CeilTimeout # noqa
ImportError: cannot import name 'CeilTimeout' from 'aiohttp.helpers' (/usr/lib/python3/dist-packages/aiohttp/helpers.py)
┌──(kali㉿kali)-[~/.local/bin]
└─$ echo $SHELL
/bin/zsh
┌──(kali㉿kali)-[~/.local/bin]
└─$ bash
┌──(kali㉿kali)-[~/.local/bin]
└─$ myip
Command 'myip' not found, did you mean:
command 'mzip' from deb mtools
Try: sudo apt install <deb name>
┌──(kali㉿kali)-[~/.local/bin]
└─$ twint -u networkchuck -s "raspberry pi"
Traceback (most recent call last):
File "/usr/local/bin/twint", line 33, in <module>
sys.exit(load_entry_point('twint==2.1.21', 'console_scripts', 'twint')())
File "/usr/local/bin/twint", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/home/kali/.local/lib/python3.9/site-packages/twint/__init__.py", line 14, in <module>
from . import run
File "/home/kali/.local/lib/python3.9/site-packages/twint/run.py", line 4, in <module>
from . import datelock, feed, get, output, verbose, storage
File "/home/kali/.local/lib/python3.9/site-packages/twint/get.py", line 12, in <module>
from aiohttp_socks import ProxyConnector, ProxyType
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/__init__.py", line 5, in <module>
from .connector import (
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/connector.py", line 8, in <module>
from aiohttp.helpers import CeilTimeout # noqa
ImportError: cannot import name 'CeilTimeout' from 'aiohttp.helpers' (/usr/lib/python3/dist-packages/aiohttp/helpers.py)
──(kali㉿kali)-[~/.local/bin]
└─$ cd ..
┌──(kali㉿kali)-[~/.local]
└─$ cd..
cd..: command not found
┌──(kali㉿kali)-[~/.local]
└─$ cd ..
┌──(kali㉿kali)-[~]
└─$
┌──(kali㉿kali)-[~]
└─$ twint -u networkchuck -s "raspberry pi"
Command 'twint' not found, did you mean:
command 'twine' from deb twine
Try: sudo apt install <deb name>
┌──(kali㉿kali)-[~/twint]
└─$ python3
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
┌──(kali㉿kali)-[~/twint]
└─$ twint -u networkchuck -s "raspberry pi"
Command 'twint' not found, did you mean:
command 'twine' from deb twine
Try: sudo apt install <deb name>
┌──(kali㉿kali)-[~]
└─$ sudo apt update && sudo apt upgrade
┌──(kali㉿kali)-[~]
└─$ systemctl reboot -i
┌──(kali㉿kali)-[~/twint]
└─$cd twint
┌──(kali㉿kali)-[~/twint]
└─$ chmod +x setup.py
┌──(kali㉿kali)-[~/twint]
└─$ python3 setup.py
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: no commands supplied
┌──(kali㉿kali)-[~/twint]
└─$ python3 setup.py --help Common commands: (see '--help-commands' for more)
setup.py build will build the package underneath 'build/'
setup.py install will install the package
┌──(kali㉿kali)-[~/twint]
└─$ sudo python3 setup.py install
running install
┌──(kali㉿kali)-[~/twint]
└─$ sudo python3 setup.py build
running build
running build_py
┌──(kali㉿kali)-[~/twint]
└─$ twint -u networkchuck -s "raspberry pi"
Traceback (most recent call last):
File "/usr/local/bin/twint", line 33, in <module>
sys.exit(load_entry_point('twint==2.1.21', 'console_scripts', 'twint')())
File "/usr/local/bin/twint", line 25, in importlib_load_entry_point
return next(matches).load()
File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
module = import_module(match.group('module'))
File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 850, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/home/kali/.local/lib/python3.9/site-packages/twint/__init__.py", line 14, in <module>
from . import run
File "/home/kali/.local/lib/python3.9/site-packages/twint/run.py", line 4, in <module>
from . import datelock, feed, get, output, verbose, storage
File "/home/kali/.local/lib/python3.9/site-packages/twint/get.py", line 12, in <module>
from aiohttp_socks import ProxyConnector, ProxyType
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/__init__.py", line 5, in <module>
from .connector import (
File "/home/kali/.local/lib/python3.9/site-packages/aiohttp_socks/connector.py", line 8, in <module>
from aiohttp.helpers import CeilTimeout # noqa
ImportError: cannot import name 'CeilTimeout' from 'aiohttp.helpers' (/usr/lib/python3/dist-packages/aiohttp/helpers.py)
Research Issue.
Fix 1: I searched on Google with the error and found,
https://github.com/twintproject/twint/issues/1071
Asked to run the command below to check if there was a .local/bin folder created.
$ echo $PATH | tr -s ":" "\n" | sort
┌──(kali㉿kali)-[~/Downloads/twint]
└─$ echo $PATH | tr -s ":" "\n" | sort 1 ⨯
/bin
/home/kali/.dotnet/tools
/home/kali/.local/bin
/sbin
/usr/bin
/usr/games
/usr/local/bin
/usr/local/games
/usr/local/sbin
/usr/sbin
Advice was to add these two lines to the .zshrc script as I am using zsh shell.
I am having the same issue in bash, so I don't think it matters at all for the moment.
$ export PYTHON_BIN_PATH="$(python3 -m site --user-base)/bin"
$ export PATH="$PATH:$PYTHON_BIN_PATH"
This method didn't work for me.
Still having the error.
Fix2: $ pip3 install yarl --force-reinstall --no-cache-dir
https://stackoverflow.com/questions/64747304/twint-python-library-is-causing-exception-for-search-query?rq=1
A fix was mentioned by a user - "I had the same issue with python3.7 and resolve it by reinstalling yarl, like this way:"
$ pip3 install yarl --force-reinstall --no-cache-dir
This method didn't work for me.
Fix3: To uninstall and reintall twint with upgrade.
https://github.com/twintproject/twint/issues/915
- pip3 uninstall twint
- pip3 install --user --upgrade git+https://github.com/twintproject/twint.git@origin/master#egg=twint
- sudo pip3 install twint
This method didn't work for me.
Fix4: Install on Google Cloud Shell.
Google Cloud New build
- sudo git clone –depth=1 https://github.com/twintproject/twint.git
- cd twint
- sudo pip3 install . -r requirements.txt
- pip3 install twint
This worked for me but once you leave the session, twint seems to be causing the same issue as I had on the Kali machine.
twint -u networkchuck –limit 20 (Did not recognise -limit 20)
twint -u networkchuck -s “raspberry pi”
New Error message.
$ twint -u networkchuck -s crypto -o rightnow.json–json
Traceback (most recent call last):
File "/home/rangersmyth_74/.local/bin/twint", line 10, in <module>
sys.exit(run_as_command())
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/cli.py", line 339, in run_as_command
main()
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/cli.py", line 330, in main
run.Search(c)
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/run.py", line 410, in Search
run(config, callback)
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/run.py", line 329, in run
get_event_loop().run_until_complete(Twint(config).main(callback))
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/run.py", line 36, in __init__
self.token.refresh()
File "/home/rangersmyth_74/.local/lib/python3.7/site-packages/twint/token.py", line 69, in refresh
raise RefreshTokenException('Could not find the Guest token in HTML')
twint.token.RefreshTokenException: Could not find the Guest token in HTML
Searching for twint.token
https://github.com/twintproject/twint/pull/1146
Fix 5. Add to PATH.
nano /etc/environment
This didn't work for me.
I have taken this from GitHub and made a new file$ nano pleasefix.py I pasted the below info into it and ran$ sudo python3 pleasefix.pyI was happy to see no error at all. BUT. $ twint -u ranger_smyth This didn't work for me. SCRIPT from >> https://github.com/twintproject/twint/pull/1146 import re
import time
import logging as logme
import requests
class TokenExpiryException(Exception):
def __init__(self, msg):
super().__init__(msg)
class RefreshTokenException(Exception):
def __init__(self, msg):
super().__init__(msg)
class Token:
def __init__(self, config):
self._session = requests.Session()
self._session.headers.update(
{'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0'})
self.config = config
self._proxies = self._set_proxies()
self._retries = 5
self._timeout = 10
self.url = 'https://twitter.com'
def _set_proxies(self) -> dict:
settings = [self.config.Proxy_type, self.config.Proxy_host, self.config.Proxy_port]
if not all(settings):
logme.debug(f"No proxy in config")
return {}
proxy_type = self.config.Proxy_type.lower()
proxy_val = f"{self.config.Proxy_host}:{self.config.Proxy_port}"
proxies = {proxy_type: proxy_val}
if proxy_type == 'http':
proxies['https'] = proxy_val
return proxies
def _request(self):
for attempt in range(self._retries + 1):
# The request is newly prepared on each retry because of potential cookie updates.
req = self._session.prepare_request(requests.Request('GET', self.url))
logme.debug(f'Retrieving {req.url}')
try:
if self._proxies:
r = self._session.send(
req,
allow_redirects=True,
timeout=self._timeout,
proxies=self._proxies,
verify=False
)
else:
r = self._session.send(req, allow_redirects=True, timeout=self._timeout)
except requests.exceptions.RequestException as exc:
if attempt < self._retries:
retrying = ', retrying'
level = logme.WARNING
else:
retrying = ''
level = logme.ERROR
logme.log(level, f'Error retrieving {req.url}: {exc!r}{retrying}')
else:
success, msg = (True, None)
msg = f': {msg}' if msg else ''
if success:
logme.debug(f'{req.url} retrieved successfully{msg}')
return r
if attempt < self._retries:
# TODO : might wanna tweak this back-off timer
sleep_time = 2.0 * 2 ** attempt
logme.info(f'Waiting {sleep_time:.0f} seconds')
time.sleep(sleep_time)
else:
msg = f'{self._retries + 1} requests to {self.url} failed, giving up.'
logme.fatal(msg)
self.config.Guest_token = None
raise RefreshTokenException(msg)
def refresh(self):
logme.debug('Retrieving guest token')
res = self._request()
match = re.search(r'\("gt=(\d+);', res.text)
if match:
logme.debug('Found guest token in HTML')
self.config.Guest_token = str(match.group(1))
else:
self.config.Guest_token = None
raise RefreshTokenException('Could not find the Guest token in HTML')
Back to the Drawing Board.
References:
https://github.com/twintproject/twint/wiki/Setup
https://github.com/twintproject/twint/wiki/Configuration
https://github.com/twintproject/twint/issues/468
https://github.com/twintproject/twint/issues/915
https://github.com/twintproject/twint/pull/917
https://github.com/twintproject/twint/pull/944
https://github.com/twintproject/twint/issues/980
https://github.com/twintproject/twint/issues/1114
https://github.com/twintproject/twint/pull/1146 - Token Fix
https://github.com/Altimis/Scweet
http://saka.docsio.net/66892325/scrape-join-dates-user-info-from-a-list-csv-of-twitter-users
http://saka.docsio.net/64747304/twint-python-library-is-causing-exception-for-search-qeruy
https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/
https://github.com/himanshudabas/twint/tree/twint-fixes
https://www.kaggle.com/zelinngilo/punya-nadia?scriptVersionId=57423357
https://www.geeksforgeeks.org/how-to-use-twint-osint-tool-on-google-cloud-console/
https://stackoverflow.com/questions/64747304/twint-python-library-is-causing-exception-for-search-query?rq=1
https://www.bountysource.com/teams/tweep/issues?tracker_ids=65007358
https://issueexplorer.com/issue/twintproject/twint/1266
How to Uninstall Package Dependencies with Pip
When you install a package with pip, it also installs all of the dependencies the package requires. Unfortunately, pip does not uninstall dependencies when you uninstall the original package. Here are a couple of different procedures that can be used to uninstall dependencies.
- If a package has been installed via a pip requirements file (i.e., pip install requirements.txt), all of the packages in requirements.txt can be uninstalled with the following command:
pip uninstall requirements.txt
- If a requirements.txt file is not available, you can use the pip show command to output all the requirements of a specified package:
pip show <packagename>
Example:
pip show cryptography
Output should be similar to:
'Requires: six, cffi'
These dependencies can then be uninstalled with the pip uninstall command. However before uninstalling, you should ensure that the packages are NOT dependencies for other existing packages.
How to Uninstall Package Dependencies with Pipenv
To uninstall all the dependencies in a Pipenv project:
- Open a command or terminal window
- cd into the project directory
pipenv uninstall --all
How to uninstall a package in Linux.
$ sudo pip3 uninstall twint
$ sudo pip uninstall -r requirements.txt -y (Do this in every directory twint is in)
$ sudo rm -r twint
$ pipenv uninstall twint
Now they are all uninstalled and also the requirements.
How do I remove a full directory in Linux?
https://www.computerhope.com/issues/ch000798.htm
rm -r mydir (mydir = twint)
rm -r twint
I had a twint folder in
1. /home/src/twint
2. /home/twint
3. /home/.local/twint
4. /home/Downloads/twint
$ sudo rm -r twint
As I have installed twint 3 different ways.
A: git clone
B: pip install
C: pipenv
Now I have to uninstall them all in case they are conflicting with each other.
What did
we learn today? We learnt how to install a GitHub file, then to install
the requirements and then to test the package with a Twitter username. We also learnt how to uninstall the package.
No comments:
Post a Comment